[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from TimestampAdvancementITest
[ RUN ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251212 03:11:54.139776 20471 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.253.254:39173
I20251212 03:11:54.140093 20471 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251212 03:11:54.140219 20471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 03:11:54.144100 20479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:11:54.144130 20477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:11:54.144325 20471 server_base.cc:1047] running on GCE node
W20251212 03:11:54.144384 20476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:11:54.144780 20471 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251212 03:11:54.144838 20471 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251212 03:11:54.144865 20471 hybrid_clock.cc:648] HybridClock initialized: now 1765509114144864 us; error 0 us; skew 500 ppm
I20251212 03:11:54.146052 20471 webserver.cc:492] Webserver started at http://127.19.253.254:37743/ using document root <none> and password file <none>
I20251212 03:11:54.146237 20471 fs_manager.cc:362] Metadata directory not provided
I20251212 03:11:54.146301 20471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 03:11:54.146378 20471 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251212 03:11:54.147459 20471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765509114134682-20471-0/minicluster-data/master-0-root/instance:
uuid: "93079d4189264fb3ba14dc72752898c2"
format_stamp: "Formatted at 2025-12-12 03:11:54 on dist-test-slave-q05w"
I20251212 03:11:54.148811 20471 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251212 03:11:54.149590 20484 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 03:11:54.149799 20471 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251212 03:11:54.149868 20471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765509114134682-20471-0/minicluster-data/master-0-root
uuid: "93079d4189264fb3ba14dc72752898c2"
format_stamp: "Formatted at 2025-12-12 03:11:54 on dist-test-slave-q05w"
I20251212 03:11:54.149936 20471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765509114134682-20471-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765509114134682-20471-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765509114134682-20471-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 03:11:54.181911 20471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 03:11:54.182253 20471 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251212 03:11:54.182374 20471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 03:11:54.183085 20490 maintenance_manager.cc:346] Maintenance manager is disabled (check --enable_maintenance_manager).
I20251212 03:11:54.186800 20471 rpc_server.cc:307] RPC server started. Bound to: 127.19.253.254:39173
I20251212 03:11:54.186874 20536 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.253.254:39173 every 8 connection(s)
I20251212 03:11:54.187446 20537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 03:11:54.189883 20537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2: Bootstrap starting.
I20251212 03:11:54.190771 20537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2: Neither blocks nor log segments found. Creating new log.
I20251212 03:11:54.191016 20537 log.cc:826] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2: Log is configured to *not* fsync() on all Append() calls
I20251212 03:11:54.191668 20537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2: No bootstrap required, opened a new log
I20251212 03:11:54.192858 20537 raft_consensus.cc:359] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93079d4189264fb3ba14dc72752898c2" member_type: VOTER }
I20251212 03:11:54.192963 20537 raft_consensus.cc:385] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 03:11:54.192996 20537 raft_consensus.cc:740] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93079d4189264fb3ba14dc72752898c2, State: Initialized, Role: FOLLOWER
I20251212 03:11:54.193122 20537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93079d4189264fb3ba14dc72752898c2" member_type: VOTER }
I20251212 03:11:54.193178 20537 raft_consensus.cc:399] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251212 03:11:54.193221 20537 raft_consensus.cc:493] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251212 03:11:54.193266 20537 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2 [term 0 FOLLOWER]: Advancing to term 1
I20251212 03:11:54.193809 20537 raft_consensus.cc:515] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93079d4189264fb3ba14dc72752898c2" member_type: VOTER }
I20251212 03:11:54.193915 20537 leader_election.cc:304] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93079d4189264fb3ba14dc72752898c2; no voters:
I20251212 03:11:54.194065 20537 leader_election.cc:290] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251212 03:11:54.194147 20540 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2 [term 1 FOLLOWER]: Leader election won for term 1
I20251212 03:11:54.194293 20537 sys_catalog.cc:565] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2 [sys.catalog]: configured and running, proceeding with master startup.
I20251212 03:11:54.194358 20540 raft_consensus.cc:697] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2 [term 1 LEADER]: Becoming Leader. State: Replica: 93079d4189264fb3ba14dc72752898c2, State: Running, Role: LEADER
I20251212 03:11:54.194495 20540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93079d4189264fb3ba14dc72752898c2" member_type: VOTER }
I20251212 03:11:54.194883 20541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93079d4189264fb3ba14dc72752898c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93079d4189264fb3ba14dc72752898c2" member_type: VOTER } }
I20251212 03:11:54.194959 20541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2 [sys.catalog]: This master's current role is: LEADER
I20251212 03:11:54.194862 20542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93079d4189264fb3ba14dc72752898c2. Latest consensus state: current_term: 1 leader_uuid: "93079d4189264fb3ba14dc72752898c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93079d4189264fb3ba14dc72752898c2" member_type: VOTER } }
I20251212 03:11:54.195102 20542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2 [sys.catalog]: This master's current role is: LEADER
I20251212 03:11:54.195362 20548 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251212 03:11:54.195734 20548 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251212 03:11:54.195920 20471 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251212 03:11:54.197275 20548 catalog_manager.cc:1357] Generated new cluster ID: c18ebfe4283e4ba489b09214ec1b9936
I20251212 03:11:54.197336 20548 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251212 03:11:54.204607 20548 catalog_manager.cc:1380] Generated new certificate authority record
I20251212 03:11:54.205199 20548 catalog_manager.cc:1514] Loading token signing keys...
I20251212 03:11:54.211930 20548 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2: Generated new TSK 0
I20251212 03:11:54.212129 20548 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251212 03:11:54.227900 20471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 03:11:54.229548 20559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:11:54.229712 20560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:11:54.229813 20471 server_base.cc:1047] running on GCE node
W20251212 03:11:54.229683 20562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:11:54.230042 20471 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251212 03:11:54.230075 20471 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251212 03:11:54.230088 20471 hybrid_clock.cc:648] HybridClock initialized: now 1765509114230089 us; error 0 us; skew 500 ppm
I20251212 03:11:54.230798 20471 webserver.cc:492] Webserver started at http://127.19.253.193:46233/ using document root <none> and password file <none>
I20251212 03:11:54.230906 20471 fs_manager.cc:362] Metadata directory not provided
I20251212 03:11:54.230960 20471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 03:11:54.231017 20471 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251212 03:11:54.231292 20471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765509114134682-20471-0/minicluster-data/ts-0-root/instance:
uuid: "e9d27948dced4eba8f8228ddc4529393"
format_stamp: "Formatted at 2025-12-12 03:11:54 on dist-test-slave-q05w"
I20251212 03:11:54.232170 20471 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251212 03:11:54.232856 20567 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 03:11:54.233026 20471 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251212 03:11:54.233085 20471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765509114134682-20471-0/minicluster-data/ts-0-root
uuid: "e9d27948dced4eba8f8228ddc4529393"
format_stamp: "Formatted at 2025-12-12 03:11:54 on dist-test-slave-q05w"
I20251212 03:11:54.233139 20471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765509114134682-20471-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765509114134682-20471-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765509114134682-20471-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 03:11:54.252174 20471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 03:11:54.252463 20471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 03:11:54.252669 20471 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251212 03:11:54.252998 20471 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251212 03:11:54.253037 20471 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251212 03:11:54.253069 20471 ts_tablet_manager.cc:616] Registered 0 tablets
I20251212 03:11:54.253098 20471 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251212 03:11:54.256899 20471 rpc_server.cc:307] RPC server started. Bound to: 127.19.253.193:46365
I20251212 03:11:54.256943 20631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.253.193:46365 every 8 connection(s)
I20251212 03:11:54.257313 20471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251212 03:11:54.259006 20471 server_base.cc:1047] running on GCE node
W20251212 03:11:54.259231 20637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:11:54.259637 20639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:11:54.262804 20635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:11:54.263023 20471 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251212 03:11:54.263082 20471 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251212 03:11:54.263104 20471 hybrid_clock.cc:648] HybridClock initialized: now 1765509114263104 us; error 0 us; skew 500 ppm
I20251212 03:11:54.263757 20471 webserver.cc:492] Webserver started at http://127.19.253.194:39405/ using document root <none> and password file <none>
I20251212 03:11:54.263862 20471 fs_manager.cc:362] Metadata directory not provided
I20251212 03:11:54.263914 20471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 03:11:54.263962 20471 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251212 03:11:54.264281 20471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765509114134682-20471-0/minicluster-data/ts-1-root/instance:
uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
format_stamp: "Formatted at 2025-12-12 03:11:54 on dist-test-slave-q05w"
I20251212 03:11:54.265404 20471 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251212 03:11:54.265995 20644 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 03:11:54.266135 20471 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251212 03:11:54.266189 20471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765509114134682-20471-0/minicluster-data/ts-1-root
uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
format_stamp: "Formatted at 2025-12-12 03:11:54 on dist-test-slave-q05w"
I20251212 03:11:54.266258 20471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765509114134682-20471-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765509114134682-20471-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765509114134682-20471-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 03:11:54.272663 20632 heartbeater.cc:344] Connected to a master server at 127.19.253.254:39173
I20251212 03:11:54.272764 20632 heartbeater.cc:461] Registering TS with master...
I20251212 03:11:54.272940 20632 heartbeater.cc:507] Master 127.19.253.254:39173 requested a full tablet report, sending...
I20251212 03:11:54.273370 20501 ts_manager.cc:194] Registered new tserver with Master: e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:11:54.274047 20501 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50686
I20251212 03:11:54.289634 20471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 03:11:54.289865 20471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 03:11:54.290076 20471 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251212 03:11:54.290473 20471 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251212 03:11:54.290519 20471 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251212 03:11:54.290550 20471 ts_tablet_manager.cc:616] Registered 0 tablets
I20251212 03:11:54.290568 20471 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251212 03:11:54.294986 20471 rpc_server.cc:307] RPC server started. Bound to: 127.19.253.194:45515
I20251212 03:11:54.295042 20709 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.253.194:45515 every 8 connection(s)
I20251212 03:11:54.295421 20471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 03:11:54.296909 20713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:11:54.297192 20471 server_base.cc:1047] running on GCE node
W20251212 03:11:54.298069 20714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:11:54.298260 20716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:11:54.298441 20471 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251212 03:11:54.298481 20471 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251212 03:11:54.298503 20471 hybrid_clock.cc:648] HybridClock initialized: now 1765509114298503 us; error 0 us; skew 500 ppm
I20251212 03:11:54.299047 20710 heartbeater.cc:344] Connected to a master server at 127.19.253.254:39173
I20251212 03:11:54.299111 20710 heartbeater.cc:461] Registering TS with master...
I20251212 03:11:54.299185 20471 webserver.cc:492] Webserver started at http://127.19.253.195:33659/ using document root <none> and password file <none>
I20251212 03:11:54.299209 20710 heartbeater.cc:507] Master 127.19.253.254:39173 requested a full tablet report, sending...
I20251212 03:11:54.299292 20471 fs_manager.cc:362] Metadata directory not provided
I20251212 03:11:54.299357 20471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 03:11:54.299407 20471 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251212 03:11:54.299449 20501 ts_manager.cc:194] Registered new tserver with Master: d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:54.299736 20471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765509114134682-20471-0/minicluster-data/ts-2-root/instance:
uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
format_stamp: "Formatted at 2025-12-12 03:11:54 on dist-test-slave-q05w"
I20251212 03:11:54.299932 20501 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50692
I20251212 03:11:54.300750 20471 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251212 03:11:54.301218 20721 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 03:11:54.301359 20471 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251212 03:11:54.301398 20471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765509114134682-20471-0/minicluster-data/ts-2-root
uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
format_stamp: "Formatted at 2025-12-12 03:11:54 on dist-test-slave-q05w"
I20251212 03:11:54.301446 20471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765509114134682-20471-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765509114134682-20471-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765509114134682-20471-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 03:11:54.309032 20471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 03:11:54.309243 20471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 03:11:54.309377 20471 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251212 03:11:54.309567 20471 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251212 03:11:54.309597 20471 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251212 03:11:54.309628 20471 ts_tablet_manager.cc:616] Registered 0 tablets
I20251212 03:11:54.309657 20471 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251212 03:11:54.312912 20471 rpc_server.cc:307] RPC server started. Bound to: 127.19.253.195:46559
I20251212 03:11:54.312996 20784 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.253.195:46559 every 8 connection(s)
I20251212 03:11:54.316069 20785 heartbeater.cc:344] Connected to a master server at 127.19.253.254:39173
I20251212 03:11:54.316126 20785 heartbeater.cc:461] Registering TS with master...
I20251212 03:11:54.316212 20785 heartbeater.cc:507] Master 127.19.253.254:39173 requested a full tablet report, sending...
I20251212 03:11:54.316423 20501 ts_manager.cc:194] Registered new tserver with Master: f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559)
I20251212 03:11:54.316426 20471 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00338305s
I20251212 03:11:54.316907 20501 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50708
I20251212 03:11:54.325397 20471 test_util.cc:276] Using random seed: -1617204848
I20251212 03:11:54.336412 20501 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50750:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251212 03:11:54.336683 20501 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251212 03:11:54.342891 20673 tablet_service.cc:1505] Processing CreateTablet for tablet 3d360069b7514dfab215af64b6929a16 (DEFAULT_TABLE table=test-workload [id=edacb51624d94a879bc1efc0287c0c8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 03:11:54.342962 20749 tablet_service.cc:1505] Processing CreateTablet for tablet 3d360069b7514dfab215af64b6929a16 (DEFAULT_TABLE table=test-workload [id=edacb51624d94a879bc1efc0287c0c8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 03:11:54.343062 20749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d360069b7514dfab215af64b6929a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 03:11:54.343377 20596 tablet_service.cc:1505] Processing CreateTablet for tablet 3d360069b7514dfab215af64b6929a16 (DEFAULT_TABLE table=test-workload [id=edacb51624d94a879bc1efc0287c0c8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 03:11:54.343472 20596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d360069b7514dfab215af64b6929a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 03:11:54.343063 20673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d360069b7514dfab215af64b6929a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 03:11:54.344614 20806 tablet_bootstrap.cc:492] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Bootstrap starting.
I20251212 03:11:54.344964 20807 tablet_bootstrap.cc:492] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Bootstrap starting.
I20251212 03:11:54.345088 20806 tablet_bootstrap.cc:654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Neither blocks nor log segments found. Creating new log.
I20251212 03:11:54.345639 20807 tablet_bootstrap.cc:654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Neither blocks nor log segments found. Creating new log.
I20251212 03:11:54.345988 20809 tablet_bootstrap.cc:492] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Bootstrap starting.
I20251212 03:11:54.346246 20807 tablet_bootstrap.cc:492] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: No bootstrap required, opened a new log
I20251212 03:11:54.346304 20807 ts_tablet_manager.cc:1403] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251212 03:11:54.346575 20809 tablet_bootstrap.cc:654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Neither blocks nor log segments found. Creating new log.
I20251212 03:11:54.346596 20807 raft_consensus.cc:359] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:54.346683 20807 raft_consensus.cc:385] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 03:11:54.346726 20807 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Initialized, Role: FOLLOWER
I20251212 03:11:54.346809 20807 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:54.346979 20807 ts_tablet_manager.cc:1434] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251212 03:11:54.347095 20632 heartbeater.cc:499] Master 127.19.253.254:39173 was elected leader, sending a full tablet report...
I20251212 03:11:54.347270 20806 tablet_bootstrap.cc:492] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: No bootstrap required, opened a new log
I20251212 03:11:54.347319 20806 ts_tablet_manager.cc:1403] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251212 03:11:54.347448 20806 raft_consensus.cc:359] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:54.347493 20806 raft_consensus.cc:385] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 03:11:54.347512 20806 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Initialized, Role: FOLLOWER
I20251212 03:11:54.347555 20806 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:54.347699 20806 ts_tablet_manager.cc:1434] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251212 03:11:54.347790 20785 heartbeater.cc:499] Master 127.19.253.254:39173 was elected leader, sending a full tablet report...
I20251212 03:11:54.350526 20809 tablet_bootstrap.cc:492] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: No bootstrap required, opened a new log
I20251212 03:11:54.350606 20809 ts_tablet_manager.cc:1403] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20251212 03:11:54.350720 20809 raft_consensus.cc:359] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:54.350770 20809 raft_consensus.cc:385] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 03:11:54.350791 20809 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Initialized, Role: FOLLOWER
I20251212 03:11:54.350845 20809 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:54.351009 20809 ts_tablet_manager.cc:1434] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251212 03:11:54.351053 20710 heartbeater.cc:499] Master 127.19.253.254:39173 was elected leader, sending a full tablet report...
I20251212 03:11:54.375200 20814 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 03:11:54.375294 20814 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:54.375492 20814 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:11:54.377321 20813 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 03:11:54.377401 20813 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:54.377948 20813 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:54.378793 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" is_pre_election: true
I20251212 03:11:54.378890 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 0.
I20251212 03:11:54.379043 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters:
I20251212 03:11:54.379135 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393" is_pre_election: true
I20251212 03:11:54.379216 20606 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 0.
I20251212 03:11:54.379139 20814 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251212 03:11:54.379432 20814 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:11:54.379472 20814 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 0 FOLLOWER]: Advancing to term 1
I20251212 03:11:54.380280 20814 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:54.380534 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:54.380621 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:54.380692 20606 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 0 FOLLOWER]: Advancing to term 1
I20251212 03:11:54.380635 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 0 FOLLOWER]: Advancing to term 1
I20251212 03:11:54.380745 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" is_pre_election: true
I20251212 03:11:54.380833 20683 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 1: Already voted for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in this term.
I20251212 03:11:54.381062 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393" is_pre_election: true
I20251212 03:11:54.381078 20814 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 1 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:11:54.381264 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:11:54.381464 20606 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 1.
I20251212 03:11:54.381541 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 1.
I20251212 03:11:54.381628 20813 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251212 03:11:54.381700 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters:
I20251212 03:11:54.381795 20814 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 1 FOLLOWER]: Leader election won for term 1
I20251212 03:11:54.381928 20814 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 1 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:11:54.381974 20814 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:54.382628 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 1 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:11:54.389307 20606 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 1 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251212 03:11:54.389433 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 1 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251212 03:11:54.389552 20818 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251212 03:11:54.389672 20814 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251212 03:11:54.405158 20471 test_util.cc:276] Using random seed: -1617125091
I20251212 03:11:54.410864 20471 test_workload.cc:405] TestWorkload: Skipping table creation because table test-workload already exists
I20251212 03:11:54.782936 20813 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:11:54.783041 20813 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 1 FOLLOWER]: Advancing to term 2
I20251212 03:11:54.783927 20813 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:54.784106 20813 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 2 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:54.784099 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 2 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 1. Current term is 2. Ops: []
I20251212 03:11:54.784262 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:54.784308 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:54.784508 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:11:54.784621 20813 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251212 03:11:54.784617 20837 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 1.40, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
W20251212 03:11:54.784762 20837 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20251212 03:11:54.784801 20837 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20251212 03:11:54.835011 20837 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 1 LEADER]: Stepping down as leader of term 1
I20251212 03:11:54.835099 20837 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:11:54.835179 20837 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 40, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:54.835310 20837 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 1 FOLLOWER]: Advancing to term 2
I20251212 03:11:54.888072 20813 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:11:54.888166 20813 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 2 FOLLOWER]: Advancing to term 3
I20251212 03:11:54.888821 20813 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:54.888996 20813 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 3 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:54.889233 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 3 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:54.889233 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 3 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:54.889325 20683 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 2 FOLLOWER]: Advancing to term 3
I20251212 03:11:54.889998 20683 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 3.
I20251212 03:11:54.890224 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:11:54.890319 20813 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 3 FOLLOWER]: Leader election won for term 3
I20251212 03:11:54.890451 20813 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 3 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:11:54.890509 20813 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 40, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:54.890997 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 1 to 3, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 3 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:11:54.903712 20857 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:11:54.903803 20857 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 1 FOLLOWER]: Advancing to term 2
I20251212 03:11:54.904460 20857 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:54.904650 20857 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 2 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:54.907186 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:54.907183 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:54.907336 20683 raft_consensus.cc:2368] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 for earlier term 2. Current term is 3.
I20251212 03:11:54.907620 20571 leader_election.cc:400] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 2 election: Vote denied by peer d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515) with higher term. Message: Invalid argument: T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 for earlier term 2. Current term is 3.
I20251212 03:11:54.907676 20571 leader_election.cc:403] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20251212 03:11:54.907759 20857 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 2 FOLLOWER]: Advancing to term 3
I20251212 03:11:54.908344 20857 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515) with higher term. Message: Invalid argument: T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 for earlier term 2. Current term is 3.
I20251212 03:11:54.983209 20606 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 3 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 3 index: 41. (index mismatch)
I20251212 03:11:54.983489 20813 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251212 03:11:54.997339 20683 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 3 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 3 index: 41. (index mismatch)
I20251212 03:11:54.997656 20813 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20251212 03:11:55.099027 20837 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:11:55.099128 20837 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 3 FOLLOWER]: Advancing to term 4
I20251212 03:11:55.099844 20837 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:55.100006 20837 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 4 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:11:55.100298 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:55.100298 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:55.100608 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:11:55.100721 20837 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251212 03:11:55.114342 20683 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 4 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 3. Current term is 4. Ops: []
I20251212 03:11:55.114717 20861 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 3.41, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251212 03:11:55.170938 20857 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:11:55.171042 20857 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 3 FOLLOWER]: Advancing to term 4
I20251212 03:11:55.171725 20857 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:55.171909 20857 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 4 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:55.172197 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:55.172197 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:55.172340 20683 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 in current term 4: Already voted for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in this term.
I20251212 03:11:55.172487 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:11:55.172609 20857 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251212 03:11:55.186107 20606 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 4 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 3. Current term is 4. Ops: []
I20251212 03:11:55.186417 20813 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 3.41, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251212 03:11:55.190117 20683 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 4 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 3. Current term is 4. Ops: []
I20251212 03:11:55.190376 20861 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 3.41, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251212 03:11:55.215133 20837 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:11:55.215232 20837 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 4 FOLLOWER]: Advancing to term 5
I20251212 03:11:55.215909 20837 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:55.216068 20837 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 5 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:11:55.216292 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 5 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:55.216308 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 5 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:55.216408 20606 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 4 FOLLOWER]: Advancing to term 5
I20251212 03:11:55.216971 20606 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 5.
I20251212 03:11:55.217155 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:11:55.217257 20837 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 5 FOLLOWER]: Leader election won for term 5
I20251212 03:11:55.217382 20837 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 5 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:11:55.217455 20837 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 3.41, Last appended by leader: 41, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:55.218097 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 3 to 5, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 5 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:11:55.228140 20854 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 3 LEADER]: Stepping down as leader of term 3
I20251212 03:11:55.228220 20854 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:11:55.228282 20854 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 3.41, Last appended by leader: 41, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:55.228402 20854 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 3 FOLLOWER]: Advancing to term 4
I20251212 03:11:55.304448 20606 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 5 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 3 index: 41. Preceding OpId from leader: term: 5 index: 42. (index mismatch)
I20251212 03:11:55.304850 20837 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251212 03:11:55.339520 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 4 FOLLOWER]: Advancing to term 5
I20251212 03:11:55.340292 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 5 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 3 index: 41. Preceding OpId from leader: term: 5 index: 42. (index mismatch)
I20251212 03:11:55.340584 20837 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251212 03:11:55.486490 20857 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:11:55.486601 20857 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 5 FOLLOWER]: Advancing to term 6
I20251212 03:11:55.487278 20857 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:55.487437 20857 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 6 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:55.487718 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:55.487718 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:55.488020 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:11:55.488165 20857 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251212 03:11:55.495332 20606 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 6 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 5. Current term is 6. Ops: []
I20251212 03:11:55.495735 20862 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 5.42, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251212 03:11:55.558043 20862 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 5 LEADER]: Stepping down as leader of term 5
I20251212 03:11:55.558147 20862 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:11:55.558267 20862 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 5.42, Last appended by leader: 42, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:55.558475 20862 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 5 FOLLOWER]: Advancing to term 6
I20251212 03:11:55.615034 20854 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:11:55.615144 20854 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 5 FOLLOWER]: Advancing to term 6
I20251212 03:11:55.615842 20854 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:55.616008 20854 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 6 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:55.616225 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:55.616246 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:55.616292 20606 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 6: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:11:55.616902 20683 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 6.
I20251212 03:11:55.617079 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:11:55.617173 20854 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 6 FOLLOWER]: Leader election won for term 6
I20251212 03:11:55.617250 20854 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 6 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:11:55.617316 20854 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 5.42, Last appended by leader: 42, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:55.617826 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 5 to 6, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 6 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:11:55.635752 20857 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:11:55.635843 20857 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 6 FOLLOWER]: Advancing to term 7
I20251212 03:11:55.636504 20857 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:55.636679 20857 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 7 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:55.636914 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 7 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:55.636914 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 7 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:55.637024 20683 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 6 FOLLOWER]: Advancing to term 7
I20251212 03:11:55.637581 20683 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 7.
I20251212 03:11:55.637826 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:11:55.637931 20857 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 7 FOLLOWER]: Leader election won for term 7
I20251212 03:11:55.638038 20857 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 7 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:11:55.638079 20857 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 5.42, Last appended by leader: 42, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:55.659817 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 6 to 7, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 7 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:11:55.728338 20606 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 7 LEADER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 6. Current term is 7. Ops: []
I20251212 03:11:55.728675 20854 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251212 03:11:55.728797 20854 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer e9d27948dced4eba8f8228ddc4529393 log is divergent from this leader: its last log entry 7.43 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 42
I20251212 03:11:55.731917 20683 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 7 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 5 index: 42. Preceding OpId from leader: term: 7 index: 43. (index mismatch)
I20251212 03:11:55.732245 20857 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251212 03:11:55.732626 20759 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 6 LEADER]: Stepping down as leader of term 6
I20251212 03:11:55.732681 20759 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:11:55.732738 20759 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 6.43, Last appended by leader: 43, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:55.732825 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 6 FOLLOWER]: Advancing to term 7
I20251212 03:11:55.733968 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 7 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 6 index: 43. Preceding OpId from leader: term: 7 index: 43. (term mismatch)
I20251212 03:11:55.734017 20759 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting all ops after (but not including) 42
I20251212 03:11:55.734059 20759 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting uncommitted NO_OP operation due to leader change: 6.43
I20251212 03:11:55.734093 20759 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:11:55.734364 20863 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20251212 03:11:55.910356 20862 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 7 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:11:55.910461 20862 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 7 FOLLOWER]: Advancing to term 8
I20251212 03:11:55.911162 20862 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:55.911438 20862 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 8 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:11:55.911554 20683 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 8 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 7. Current term is 8. Ops: []
I20251212 03:11:55.911597 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 8 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:55.911617 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 8 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:55.911756 20865 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 7.43, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20251212 03:11:55.911868 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:11:55.911957 20862 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251212 03:11:56.015620 20683 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 8 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 7. Current term is 8. Ops: []
I20251212 03:11:56.015916 20863 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 7.43, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20251212 03:11:56.046746 20857 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 7 LEADER]: Stepping down as leader of term 7
I20251212 03:11:56.046850 20857 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:11:56.046927 20857 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 7.43, Last appended by leader: 43, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:56.047052 20857 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 7 FOLLOWER]: Advancing to term 8
I20251212 03:11:56.047381 20862 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:11:56.047449 20862 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 8 FOLLOWER]: Advancing to term 9
I20251212 03:11:56.048161 20862 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:56.048303 20862 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 9 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:11:56.048519 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 9 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:56.048519 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 9 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:56.048645 20606 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 8 FOLLOWER]: Advancing to term 9
I20251212 03:11:56.049202 20606 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 9.
I20251212 03:11:56.049386 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:11:56.049468 20862 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 9 FOLLOWER]: Leader election won for term 9
I20251212 03:11:56.049574 20862 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 9 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:11:56.049620 20862 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 7.43, Last appended by leader: 43, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:56.050184 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 7 to 9, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 9 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:11:56.050933 20813 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:11:56.050992 20813 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 7 FOLLOWER]: Advancing to term 8
I20251212 03:11:56.051503 20813 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:56.051642 20813 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 8 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:56.051802 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 8 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:56.051832 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 8 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:56.051869 20606 raft_consensus.cc:2368] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 8. Current term is 9.
I20251212 03:11:56.052016 20723 leader_election.cc:400] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 8 election: Vote denied by peer e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365) with higher term. Message: Invalid argument: T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 8. Current term is 9.
I20251212 03:11:56.052062 20723 leader_election.cc:403] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20251212 03:11:56.052134 20813 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 8 FOLLOWER]: Advancing to term 9
I20251212 03:11:56.052613 20813 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365) with higher term. Message: Invalid argument: T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 8. Current term is 9.
I20251212 03:11:56.145684 20606 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 9 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 7 index: 43. Preceding OpId from leader: term: 9 index: 44. (index mismatch)
I20251212 03:11:56.146062 20862 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20251212 03:11:56.152894 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 9 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 7 index: 43. Preceding OpId from leader: term: 9 index: 44. (index mismatch)
I20251212 03:11:56.153170 20862 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20251212 03:11:56.331525 20813 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:11:56.331645 20813 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 9 FOLLOWER]: Advancing to term 10
I20251212 03:11:56.332350 20813 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:56.332525 20813 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 10 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:56.332741 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 10 candidate_status { last_received { term: 9 index: 44 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:56.332757 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 10 candidate_status { last_received { term: 9 index: 44 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:56.332962 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:11:56.333103 20813 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20251212 03:11:56.348033 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 10 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 9. Current term is 10. Ops: []
I20251212 03:11:56.348325 20866 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 9.44, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20251212 03:11:56.383615 20866 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 9 LEADER]: Stepping down as leader of term 9
I20251212 03:11:56.383723 20866 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:11:56.383810 20866 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 9.44, Last appended by leader: 44, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:56.383939 20866 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 9 FOLLOWER]: Advancing to term 10
I20251212 03:11:56.442584 20813 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:11:56.442698 20813 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 10 FOLLOWER]: Advancing to term 11
I20251212 03:11:56.443380 20813 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:56.443554 20813 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 11 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:56.443786 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 11 candidate_status { last_received { term: 9 index: 44 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:56.443796 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 11 candidate_status { last_received { term: 9 index: 44 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:56.443881 20683 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 10 FOLLOWER]: Advancing to term 11
I20251212 03:11:56.444566 20683 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 11.
I20251212 03:11:56.444785 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:11:56.444881 20813 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 11 FOLLOWER]: Leader election won for term 11
I20251212 03:11:56.444976 20813 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 11 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:11:56.445017 20813 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 9.44, Last appended by leader: 44, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:56.445541 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 9 to 11, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 11 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:11:56.459010 20857 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:11:56.459111 20857 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 9 FOLLOWER]: Advancing to term 10
I20251212 03:11:56.459777 20857 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:56.459960 20857 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 10 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:56.460191 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 10 candidate_status { last_received { term: 9 index: 44 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:56.460186 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 10 candidate_status { last_received { term: 9 index: 44 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:56.460276 20683 raft_consensus.cc:2368] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 for earlier term 10. Current term is 11.
I20251212 03:11:56.460511 20571 leader_election.cc:400] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 10 election: Vote denied by peer d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515) with higher term. Message: Invalid argument: T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 for earlier term 10. Current term is 11.
I20251212 03:11:56.460546 20571 leader_election.cc:403] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20251212 03:11:56.460637 20857 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 10 FOLLOWER]: Advancing to term 11
I20251212 03:11:56.461158 20857 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515) with higher term. Message: Invalid argument: T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 for earlier term 10. Current term is 11.
I20251212 03:11:56.468169 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:11:56.469413 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:11:56.471618 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:11:56.475819 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:11:56.484009 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:11:56.500221 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:11:56.532475 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:11:56.537065 20606 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 11 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 9 index: 44. Preceding OpId from leader: term: 11 index: 45. (index mismatch)
I20251212 03:11:56.537349 20813 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20251212 03:11:56.538893 20683 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 11 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 9 index: 44. Preceding OpId from leader: term: 11 index: 45. (index mismatch)
I20251212 03:11:56.539126 20813 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20251212 03:11:56.596755 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:11:56.640491 20866 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 11 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:11:56.640586 20866 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 11 FOLLOWER]: Advancing to term 12
I20251212 03:11:56.646589 20866 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:56.646832 20866 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 12 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:11:56.647015 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 12 candidate_status { last_received { term: 11 index: 45 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:56.647075 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 12 candidate_status { last_received { term: 11 index: 45 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:56.647292 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:11:56.647383 20866 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20251212 03:11:56.650530 20683 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 12 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 11. Current term is 12. Ops: []
I20251212 03:11:56.650764 20813 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 11.45, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20251212 03:11:56.650859 20813 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 11 LEADER]: Stepping down as leader of term 11
I20251212 03:11:56.650895 20813 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:11:56.650954 20813 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 11.45, Last appended by leader: 45, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:56.651029 20813 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 11 FOLLOWER]: Advancing to term 12
I20251212 03:11:56.725032 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:11:56.731333 20857 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 11 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:11:56.731434 20857 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 11 FOLLOWER]: Advancing to term 12
I20251212 03:11:56.732103 20857 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:56.732281 20857 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 12 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:56.732520 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 12 candidate_status { last_received { term: 11 index: 45 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:56.732555 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 12 candidate_status { last_received { term: 11 index: 45 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:56.732669 20683 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 in current term 12: Already voted for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in this term.
I20251212 03:11:56.733163 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 12.
I20251212 03:11:56.733340 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:11:56.733430 20857 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 12 FOLLOWER]: Leader election won for term 12
I20251212 03:11:56.733531 20857 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 12 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:11:56.733574 20857 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 11.45, Last appended by leader: 45, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:56.734032 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 11 to 12, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 12 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:11:56.775985 20866 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:11:56.776093 20866 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 12 FOLLOWER]: Advancing to term 13
I20251212 03:11:56.776809 20866 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:56.776993 20866 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 13 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:11:56.777223 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 13 candidate_status { last_received { term: 11 index: 45 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:56.777223 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 13 candidate_status { last_received { term: 11 index: 45 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:56.777323 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 12 FOLLOWER]: Advancing to term 13
I20251212 03:11:56.777911 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 13.
I20251212 03:11:56.778116 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:11:56.778229 20866 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 13 FOLLOWER]: Leader election won for term 13
I20251212 03:11:56.778290 20866 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 13 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:11:56.778352 20866 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 11.45, Last appended by leader: 45, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:56.778975 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 12 to 13, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 13 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:11:56.817420 20683 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 13 LEADER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 12. Current term is 13. Ops: []
I20251212 03:11:56.817730 20857 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20251212 03:11:56.817844 20857 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer d3fe965dfda94fd4bf3f61a5c0452d63 log is divergent from this leader: its last log entry 13.46 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 45
I20251212 03:11:56.817886 20869 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 12 LEADER]: Stepping down as leader of term 12
I20251212 03:11:56.817941 20869 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:11:56.818116 20869 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 12.46, Last appended by leader: 46, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:56.818269 20869 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 12 FOLLOWER]: Advancing to term 13
I20251212 03:11:56.855343 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 13 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 11 index: 45. Preceding OpId from leader: term: 13 index: 46. (index mismatch)
I20251212 03:11:56.855684 20866 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20251212 03:11:56.856899 20606 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 13 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 12 index: 46. Preceding OpId from leader: term: 13 index: 46. (term mismatch)
I20251212 03:11:56.856963 20606 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting all ops after (but not including) 45
I20251212 03:11:56.856986 20606 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting uncommitted NO_OP operation due to leader change: 12.46
I20251212 03:11:56.857010 20606 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:11:56.857311 20866 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20251212 03:11:56.981321 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:11:57.130306 20813 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 13 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:11:57.130403 20813 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 13 FOLLOWER]: Advancing to term 14
I20251212 03:11:57.131114 20813 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:57.131275 20813 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 14 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:57.131485 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 14 candidate_status { last_received { term: 13 index: 46 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:57.131492 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 14 candidate_status { last_received { term: 13 index: 46 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:57.131727 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:11:57.131824 20813 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20251212 03:11:57.144212 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 14 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 13. Current term is 14. Ops: []
I20251212 03:11:57.144464 20867 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 13.46, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20251212 03:11:57.144594 20867 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 13 LEADER]: Stepping down as leader of term 13
I20251212 03:11:57.144639 20867 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:11:57.144709 20867 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 13.46, Last appended by leader: 46, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:57.144788 20867 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 13 FOLLOWER]: Advancing to term 14
I20251212 03:11:57.227133 20869 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 13 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:11:57.227233 20869 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 13 FOLLOWER]: Advancing to term 14
I20251212 03:11:57.227936 20869 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:57.228195 20869 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 14 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:57.228335 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 14 candidate_status { last_received { term: 13 index: 46 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:57.228416 20759 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 in current term 14: Already voted for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in this term.
I20251212 03:11:57.228346 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 14 candidate_status { last_received { term: 13 index: 46 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:57.229125 20683 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 14.
I20251212 03:11:57.229362 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:11:57.229461 20869 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 14 FOLLOWER]: Leader election won for term 14
I20251212 03:11:57.229522 20869 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 14 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:11:57.229617 20869 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 13.46, Last appended by leader: 46, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:57.230091 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 13 to 14, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 14 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:11:57.252305 20813 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:11:57.252408 20813 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 14 FOLLOWER]: Advancing to term 15
I20251212 03:11:57.253142 20813 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:57.253306 20813 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 15 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:57.253561 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 15 candidate_status { last_received { term: 13 index: 46 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:57.253561 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 15 candidate_status { last_received { term: 13 index: 46 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:57.253679 20683 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 14 FOLLOWER]: Advancing to term 15
I20251212 03:11:57.254290 20683 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 15.
I20251212 03:11:57.254462 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:11:57.254571 20813 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 15 FOLLOWER]: Leader election won for term 15
I20251212 03:11:57.254685 20813 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 15 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:11:57.254748 20813 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 13.46, Last appended by leader: 46, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:57.255195 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 14 to 15, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 15 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:11:57.312454 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 15 LEADER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 14. Current term is 15. Ops: []
I20251212 03:11:57.312744 20869 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20251212 03:11:57.312841 20869 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer f894b0d567ab4a2ebdcb416d04ef48e5 log is divergent from this leader: its last log entry 15.47 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 46
I20251212 03:11:57.312876 20857 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 14 LEADER]: Stepping down as leader of term 14
I20251212 03:11:57.312943 20857 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:11:57.313032 20857 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 14.47, Last appended by leader: 47, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:57.313107 20857 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 14 FOLLOWER]: Advancing to term 15
I20251212 03:11:57.365806 20606 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 15 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 14 index: 47. Preceding OpId from leader: term: 15 index: 47. (term mismatch)
I20251212 03:11:57.365887 20606 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting all ops after (but not including) 46
I20251212 03:11:57.365901 20606 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting uncommitted NO_OP operation due to leader change: 14.47
I20251212 03:11:57.365916 20606 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:11:57.366179 20813 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20251212 03:11:57.367482 20683 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 15 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 13 index: 46. Preceding OpId from leader: term: 15 index: 47. (index mismatch)
I20251212 03:11:57.367738 20813 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20251212 03:11:57.468676 20867 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 15 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:11:57.468791 20867 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 15 FOLLOWER]: Advancing to term 16
I20251212 03:11:57.469513 20867 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:57.469691 20867 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 16 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:11:57.469934 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 16 candidate_status { last_received { term: 15 index: 47 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:57.469944 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 16 candidate_status { last_received { term: 15 index: 47 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:57.470155 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:11:57.470310 20867 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20251212 03:11:57.484819 20683 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 16 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 15. Current term is 16. Ops: []
I20251212 03:11:57.485072 20813 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 15.47, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20251212 03:11:57.485181 20813 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 15 LEADER]: Stepping down as leader of term 15
I20251212 03:11:57.485201 20813 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:11:57.485253 20813 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 15.47, Last appended by leader: 47, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:57.485335 20813 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 15 FOLLOWER]: Advancing to term 16
I20251212 03:11:57.493584 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:11:57.556632 20857 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 15 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:11:57.556747 20857 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 15 FOLLOWER]: Advancing to term 16
I20251212 03:11:57.557384 20857 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:57.557566 20857 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 16 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:57.557852 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 16 candidate_status { last_received { term: 15 index: 47 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:57.557852 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 16 candidate_status { last_received { term: 15 index: 47 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:57.557981 20683 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 in current term 16: Already voted for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in this term.
I20251212 03:11:57.558591 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 16.
I20251212 03:11:57.558810 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:11:57.558890 20857 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 16 FOLLOWER]: Leader election won for term 16
I20251212 03:11:57.558945 20857 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 16 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:11:57.558997 20857 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 15.47, Last appended by leader: 47, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:57.559437 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 15 to 16, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 16 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:11:57.585779 20867 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:11:57.585883 20867 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 16 FOLLOWER]: Advancing to term 17
I20251212 03:11:57.586622 20867 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:57.586805 20867 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 17 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:11:57.587023 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 17 candidate_status { last_received { term: 15 index: 47 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:57.587026 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 17 candidate_status { last_received { term: 15 index: 47 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:57.587119 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 16 FOLLOWER]: Advancing to term 17
I20251212 03:11:57.587693 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 17.
I20251212 03:11:57.587882 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:11:57.587975 20867 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 17 FOLLOWER]: Leader election won for term 17
I20251212 03:11:57.588033 20867 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 17 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:11:57.588086 20867 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 15.47, Last appended by leader: 47, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:57.588589 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 16 to 17, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 17 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:11:57.634864 20683 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 17 LEADER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 16. Current term is 17. Ops: []
I20251212 03:11:57.635195 20857 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20251212 03:11:57.635319 20857 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer d3fe965dfda94fd4bf3f61a5c0452d63 log is divergent from this leader: its last log entry 17.48 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 47
I20251212 03:11:57.635360 20869 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 16 LEADER]: Stepping down as leader of term 16
I20251212 03:11:57.635419 20869 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:11:57.635484 20869 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 16.48, Last appended by leader: 48, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:57.635576 20869 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 16 FOLLOWER]: Advancing to term 17
I20251212 03:11:57.689994 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 17 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 15 index: 47. Preceding OpId from leader: term: 17 index: 48. (index mismatch)
I20251212 03:11:57.690323 20867 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20251212 03:11:57.691385 20606 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 17 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 16 index: 48. Preceding OpId from leader: term: 17 index: 48. (term mismatch)
I20251212 03:11:57.691452 20606 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting all ops after (but not including) 47
I20251212 03:11:57.691475 20606 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting uncommitted NO_OP operation due to leader change: 16.48
I20251212 03:11:57.691493 20606 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:11:57.691733 20871 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20251212 03:11:57.899473 20869 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 17 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:11:57.899647 20869 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 17 FOLLOWER]: Advancing to term 18
I20251212 03:11:57.900461 20869 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:57.900641 20606 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 18 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 17. Current term is 18. Ops: []
I20251212 03:11:57.900741 20869 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 18 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:57.900846 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 18 candidate_status { last_received { term: 17 index: 48 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:57.900889 20867 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 17.48, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20251212 03:11:57.900962 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 18 candidate_status { last_received { term: 17 index: 48 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:57.901010 20867 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 17 LEADER]: Stepping down as leader of term 17
I20251212 03:11:57.901049 20867 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:11:57.901120 20867 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 17.48, Last appended by leader: 48, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:57.901247 20867 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 17 FOLLOWER]: Advancing to term 18
I20251212 03:11:57.901124 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:11:57.901330 20869 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20251212 03:11:57.968357 20813 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 17 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:11:57.968458 20813 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 17 FOLLOWER]: Advancing to term 18
I20251212 03:11:57.969153 20813 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:57.969331 20813 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 18 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:57.969570 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 18 candidate_status { last_received { term: 17 index: 48 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:57.969666 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 18 candidate_status { last_received { term: 17 index: 48 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:57.969903 20606 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 18: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:11:57.970196 20683 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 18.
I20251212 03:11:57.970415 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:11:57.970563 20813 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 18 FOLLOWER]: Leader election won for term 18
I20251212 03:11:57.970623 20813 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 18 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:11:57.970674 20813 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 17.48, Last appended by leader: 48, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:57.971125 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 17 to 18, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 18 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:11:58.002297 20869 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:11:58.002414 20869 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 18 FOLLOWER]: Advancing to term 19
I20251212 03:11:58.003198 20869 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:58.003475 20869 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 19 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:58.003575 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 19 candidate_status { last_received { term: 17 index: 48 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:58.003575 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 19 candidate_status { last_received { term: 17 index: 48 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:58.003760 20683 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 18 FOLLOWER]: Advancing to term 19
I20251212 03:11:58.004350 20683 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 19.
I20251212 03:11:58.004546 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:11:58.004675 20869 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 19 FOLLOWER]: Leader election won for term 19
I20251212 03:11:58.004746 20869 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 19 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:11:58.004896 20869 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 17.48, Last appended by leader: 48, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:58.005342 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 18 to 19, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 19 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:11:58.059816 20683 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 19 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 18. Current term is 19. Ops: []
I20251212 03:11:58.060191 20813 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20251212 03:11:58.060297 20813 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 18 LEADER]: Stepping down as leader of term 18
I20251212 03:11:58.060328 20813 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:11:58.060400 20813 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 18.49, Last appended by leader: 49, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:58.060465 20813 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 18 FOLLOWER]: Advancing to term 19
I20251212 03:11:58.103878 20683 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 19 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 17 index: 48. Preceding OpId from leader: term: 19 index: 49. (index mismatch)
I20251212 03:11:58.104213 20869 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20251212 03:11:58.105372 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 19 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 18 index: 49. Preceding OpId from leader: term: 19 index: 49. (term mismatch)
I20251212 03:11:58.105444 20759 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting all ops after (but not including) 48
I20251212 03:11:58.105530 20759 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting uncommitted NO_OP operation due to leader change: 18.49
I20251212 03:11:58.105566 20759 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:11:58.105765 20872 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20251212 03:11:58.205919 20867 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 19 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:11:58.206023 20867 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 19 FOLLOWER]: Advancing to term 20
I20251212 03:11:58.206768 20867 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:58.206951 20867 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 20 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:11:58.207183 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 20 candidate_status { last_received { term: 19 index: 49 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:58.207183 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 20 candidate_status { last_received { term: 19 index: 49 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:58.207327 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 19 FOLLOWER]: Advancing to term 20
I20251212 03:11:58.207901 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 20.
I20251212 03:11:58.208079 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:11:58.208178 20867 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 20 FOLLOWER]: Leader election won for term 20
I20251212 03:11:58.208225 20867 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 20 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:11:58.208274 20867 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 19.49, Last appended by leader: 49, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:58.208797 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 19 to 20, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 20 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:11:58.209776 20683 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 20 LEADER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 19. Current term is 20. Ops: []
I20251212 03:11:58.209985 20872 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 19.49, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20251212 03:11:58.210083 20872 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer d3fe965dfda94fd4bf3f61a5c0452d63 log is divergent from this leader: its last log entry 20.50 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 49
I20251212 03:11:58.210114 20869 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 19 LEADER]: Stepping down as leader of term 19
I20251212 03:11:58.210162 20869 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:11:58.210251 20869 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 19.49, Last appended by leader: 49, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:58.210323 20869 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 19 FOLLOWER]: Advancing to term 20
I20251212 03:11:58.302842 20606 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 20 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 19 index: 49. Preceding OpId from leader: term: 20 index: 50. (index mismatch)
I20251212 03:11:58.303164 20867 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20251212 03:11:58.304476 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 20 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 19 index: 49. Preceding OpId from leader: term: 20 index: 50. (index mismatch)
I20251212 03:11:58.304665 20873 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20251212 03:11:58.493808 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:11:58.495364 20869 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 20 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:11:58.495448 20869 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 20 FOLLOWER]: Advancing to term 21
I20251212 03:11:58.496279 20869 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:58.496555 20869 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 21 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:58.496663 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 21 candidate_status { last_received { term: 20 index: 50 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:58.496667 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 21 candidate_status { last_received { term: 20 index: 50 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:58.496927 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:11:58.497041 20869 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20251212 03:11:58.505388 20606 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 21 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 20. Current term is 21. Ops: []
I20251212 03:11:58.505707 20866 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 20.50, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20251212 03:11:58.505836 20866 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 20 LEADER]: Stepping down as leader of term 20
I20251212 03:11:58.505872 20866 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:11:58.505932 20866 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 20.50, Last appended by leader: 50, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:58.506003 20866 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 20 FOLLOWER]: Advancing to term 21
I20251212 03:11:58.570024 20870 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 20 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:11:58.570128 20870 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 20 FOLLOWER]: Advancing to term 21
I20251212 03:11:58.570873 20870 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:58.571025 20870 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 21 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:58.571276 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 21 candidate_status { last_received { term: 20 index: 50 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:58.571280 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 21 candidate_status { last_received { term: 20 index: 50 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:58.571398 20606 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 21: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:11:58.571942 20683 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 21.
I20251212 03:11:58.572129 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:11:58.572212 20870 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 21 FOLLOWER]: Leader election won for term 21
I20251212 03:11:58.572261 20870 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 21 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:11:58.572317 20870 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 20.50, Last appended by leader: 50, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:58.572721 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 20 to 21, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 21 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:11:58.627507 20869 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:11:58.627619 20869 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 21 FOLLOWER]: Advancing to term 22
I20251212 03:11:58.628244 20869 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:58.628397 20869 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 22 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:58.628662 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 22 candidate_status { last_received { term: 20 index: 50 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:58.628732 20683 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 21 FOLLOWER]: Advancing to term 22
I20251212 03:11:58.628691 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 22 candidate_status { last_received { term: 20 index: 50 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:58.629307 20683 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 22.
I20251212 03:11:58.629501 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:11:58.629602 20869 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 22 FOLLOWER]: Leader election won for term 22
I20251212 03:11:58.629657 20869 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 22 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:11:58.629714 20869 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 20.50, Last appended by leader: 50, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:58.630223 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 21 to 22, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 22 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:11:58.666231 20606 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 22 LEADER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 21. Current term is 22. Ops: []
I20251212 03:11:58.666496 20870 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20251212 03:11:58.666641 20870 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer e9d27948dced4eba8f8228ddc4529393 log is divergent from this leader: its last log entry 22.51 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 50
I20251212 03:11:58.666688 20813 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 21 LEADER]: Stepping down as leader of term 21
I20251212 03:11:58.666754 20813 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:11:58.666823 20813 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 21.51, Last appended by leader: 51, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:58.666919 20813 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 21 FOLLOWER]: Advancing to term 22
I20251212 03:11:58.710769 20683 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 22 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 20 index: 50. Preceding OpId from leader: term: 22 index: 51. (index mismatch)
I20251212 03:11:58.711134 20869 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20251212 03:11:58.712241 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 22 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 21 index: 51. Preceding OpId from leader: term: 22 index: 51. (term mismatch)
I20251212 03:11:58.712291 20759 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting all ops after (but not including) 50
I20251212 03:11:58.712309 20759 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting uncommitted NO_OP operation due to leader change: 21.51
I20251212 03:11:58.712333 20759 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:11:58.712500 20869 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20251212 03:11:58.936740 20870 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 22 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:11:58.936842 20870 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 22 FOLLOWER]: Advancing to term 23
I20251212 03:11:58.937510 20870 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:58.937666 20870 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 23 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:58.937880 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 23 candidate_status { last_received { term: 22 index: 51 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:58.937885 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 23 candidate_status { last_received { term: 22 index: 51 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:58.938190 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:11:58.938314 20870 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20251212 03:11:58.942516 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 23 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 22. Current term is 23. Ops: []
I20251212 03:11:58.942842 20874 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 22.51, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20251212 03:11:58.942952 20874 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 22 LEADER]: Stepping down as leader of term 22
I20251212 03:11:58.942973 20874 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:11:58.943043 20874 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 22.51, Last appended by leader: 51, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:58.943117 20874 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 22 FOLLOWER]: Advancing to term 23
I20251212 03:11:58.992563 20866 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 22 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:11:58.992669 20866 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 22 FOLLOWER]: Advancing to term 23
I20251212 03:11:58.993461 20866 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:58.993634 20866 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 23 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:11:58.993882 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 23 candidate_status { last_received { term: 22 index: 51 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:58.993975 20759 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate d3fe965dfda94fd4bf3f61a5c0452d63 in current term 23: Already voted for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in this term.
I20251212 03:11:58.993885 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 23 candidate_status { last_received { term: 22 index: 51 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:58.994647 20606 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 23.
I20251212 03:11:58.994865 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:11:58.994956 20866 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 23 FOLLOWER]: Leader election won for term 23
I20251212 03:11:58.995007 20866 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 23 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:11:58.995065 20866 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 22.51, Last appended by leader: 51, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:58.995541 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 22 to 23, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 23 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:11:59.040673 20870 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:11:59.040794 20870 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 23 FOLLOWER]: Advancing to term 24
I20251212 03:11:59.041446 20870 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:59.041610 20870 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 24 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:59.041857 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 24 candidate_status { last_received { term: 22 index: 51 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:59.041857 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 24 candidate_status { last_received { term: 22 index: 51 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:59.041996 20606 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 23 FOLLOWER]: Advancing to term 24
I20251212 03:11:59.042629 20606 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 24.
I20251212 03:11:59.042809 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:11:59.042905 20870 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 24 FOLLOWER]: Leader election won for term 24
I20251212 03:11:59.042974 20870 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 24 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:11:59.043026 20870 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 22.51, Last appended by leader: 51, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:59.043612 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 23 to 24, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 24 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:11:59.099938 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 24 LEADER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 23. Current term is 24. Ops: []
I20251212 03:11:59.100296 20866 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20251212 03:11:59.100425 20866 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer f894b0d567ab4a2ebdcb416d04ef48e5 log is divergent from this leader: its last log entry 24.52 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 51
I20251212 03:11:59.100529 20873 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 23 LEADER]: Stepping down as leader of term 23
I20251212 03:11:59.100600 20873 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:11:59.100725 20873 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 23.52, Last appended by leader: 52, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:59.100839 20873 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 23 FOLLOWER]: Advancing to term 24
I20251212 03:11:59.122004 20606 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 24 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 22 index: 51. Preceding OpId from leader: term: 24 index: 52. (index mismatch)
I20251212 03:11:59.122370 20870 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20251212 03:11:59.123566 20683 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 24 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 23 index: 52. Preceding OpId from leader: term: 24 index: 52. (term mismatch)
I20251212 03:11:59.123632 20683 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting all ops after (but not including) 51
I20251212 03:11:59.123653 20683 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting uncommitted NO_OP operation due to leader change: 23.52
I20251212 03:11:59.123675 20683 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:11:59.123871 20813 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20251212 03:11:59.224409 20874 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 24 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:11:59.224526 20874 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 24 FOLLOWER]: Advancing to term 25
I20251212 03:11:59.225210 20874 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:59.225384 20874 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 25 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:59.225621 20683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 25 candidate_status { last_received { term: 24 index: 52 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:59.225643 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 25 candidate_status { last_received { term: 24 index: 52 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:59.225731 20683 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 24 FOLLOWER]: Advancing to term 25
I20251212 03:11:59.226491 20683 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 25.
I20251212 03:11:59.226684 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 25 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 24. Current term is 25. Ops: []
I20251212 03:11:59.226727 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:11:59.226835 20874 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 25 FOLLOWER]: Leader election won for term 25
I20251212 03:11:59.226893 20874 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 25 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:11:59.226871 20875 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 24.52, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20251212 03:11:59.226949 20875 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 24 LEADER]: Stepping down as leader of term 24
I20251212 03:11:59.226990 20875 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:11:59.226955 20874 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 24.52, Last appended by leader: 52, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:59.227046 20875 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 24.52, Last appended by leader: 52, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:59.227142 20875 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 24 FOLLOWER]: Advancing to term 25
I20251212 03:11:59.227380 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 24 to 25, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 25 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:11:59.327867 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 25 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 24 index: 52. Preceding OpId from leader: term: 25 index: 53. (index mismatch)
I20251212 03:11:59.328173 20874 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20251212 03:11:59.329183 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 25 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 24 index: 52. Preceding OpId from leader: term: 25 index: 53. (index mismatch)
I20251212 03:11:59.329484 20869 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20251212 03:11:59.430249 20873 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 25 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:11:59.430380 20873 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 25 FOLLOWER]: Advancing to term 26
I20251212 03:11:59.431321 20873 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:59.431453 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 26 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 25. Current term is 26. Ops: []
I20251212 03:11:59.431569 20873 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 26 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:11:59.431655 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 26 candidate_status { last_received { term: 25 index: 53 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:59.431699 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 26 candidate_status { last_received { term: 25 index: 53 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:59.431713 20869 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 25.53, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20251212 03:11:59.431893 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:11:59.431910 20869 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 25 LEADER]: Stepping down as leader of term 25
I20251212 03:11:59.431965 20869 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:11:59.432025 20869 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 25.53, Last appended by leader: 53, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:59.432024 20873 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20251212 03:11:59.432112 20869 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 25 FOLLOWER]: Advancing to term 26
I20251212 03:11:59.494117 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:11:59.517761 20875 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 25 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:11:59.517863 20875 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 25 FOLLOWER]: Advancing to term 26
I20251212 03:11:59.518574 20875 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:59.518743 20875 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 26 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:59.518961 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 26 candidate_status { last_received { term: 25 index: 53 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:59.518973 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 26 candidate_status { last_received { term: 25 index: 53 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:59.519088 20682 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 26: Already voted for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in this term.
I20251212 03:11:59.519704 20606 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 26.
I20251212 03:11:59.519918 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:11:59.520038 20875 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 26 FOLLOWER]: Leader election won for term 26
I20251212 03:11:59.520105 20875 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 26 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:11:59.520153 20875 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 25.53, Last appended by leader: 53, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:59.520677 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 25 to 26, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 26 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:11:59.565532 20873 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:11:59.565663 20873 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 26 FOLLOWER]: Advancing to term 27
I20251212 03:11:59.566351 20873 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:59.566514 20873 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 27 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:11:59.566768 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 27 candidate_status { last_received { term: 25 index: 53 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:59.566768 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 27 candidate_status { last_received { term: 25 index: 53 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:59.566895 20606 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 26 FOLLOWER]: Advancing to term 27
I20251212 03:11:59.567550 20606 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 27.
I20251212 03:11:59.567762 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:11:59.567873 20873 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 27 FOLLOWER]: Leader election won for term 27
I20251212 03:11:59.567950 20873 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 27 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:11:59.568015 20873 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 25.53, Last appended by leader: 53, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:59.568431 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 26 to 27, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 27 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:11:59.623787 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 27 LEADER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 26. Current term is 27. Ops: []
I20251212 03:11:59.624054 20875 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20251212 03:11:59.624161 20875 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer d3fe965dfda94fd4bf3f61a5c0452d63 log is divergent from this leader: its last log entry 27.54 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 53
I20251212 03:11:59.624186 20875 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 26 LEADER]: Stepping down as leader of term 26
I20251212 03:11:59.624199 20875 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:11:59.624249 20875 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 26.54, Last appended by leader: 54, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:59.624327 20875 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 26 FOLLOWER]: Advancing to term 27
I20251212 03:11:59.656829 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 27 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 26 index: 54. Preceding OpId from leader: term: 27 index: 54. (term mismatch)
I20251212 03:11:59.656911 20759 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting all ops after (but not including) 53
I20251212 03:11:59.656925 20759 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting uncommitted NO_OP operation due to leader change: 26.54
I20251212 03:11:59.656940 20759 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:11:59.657191 20873 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20251212 03:11:59.658327 20606 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 27 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 25 index: 53. Preceding OpId from leader: term: 27 index: 54. (index mismatch)
I20251212 03:11:59.658555 20876 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20251212 03:11:59.881140 20875 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 27 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:11:59.881244 20875 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 27 FOLLOWER]: Advancing to term 28
I20251212 03:11:59.882001 20875 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:59.882136 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 28 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 27. Current term is 28. Ops: []
I20251212 03:11:59.882275 20875 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 28 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:59.882347 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 28 candidate_status { last_received { term: 27 index: 54 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:59.882405 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 28 candidate_status { last_received { term: 27 index: 54 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:59.882411 20866 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 27.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20251212 03:11:59.882625 20866 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 27 LEADER]: Stepping down as leader of term 27
I20251212 03:11:59.882652 20866 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:11:59.882725 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:11:59.882733 20866 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 27.54, Last appended by leader: 54, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:59.882838 20875 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20251212 03:11:59.882841 20866 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 27 FOLLOWER]: Advancing to term 28
I20251212 03:11:59.922138 20869 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 27 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:11:59.922286 20869 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 27 FOLLOWER]: Advancing to term 28
I20251212 03:11:59.922959 20869 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:59.923130 20869 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 28 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:59.923314 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 28 candidate_status { last_received { term: 27 index: 54 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:11:59.923386 20759 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 in current term 28: Already voted for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in this term.
I20251212 03:11:59.923373 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 28 candidate_status { last_received { term: 27 index: 54 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:59.924273 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 28.
I20251212 03:11:59.926273 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:11:59.926369 20869 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 28 FOLLOWER]: Leader election won for term 28
I20251212 03:11:59.926465 20869 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 28 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:11:59.926523 20869 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 27.54, Last appended by leader: 54, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:59.927088 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 27 to 28, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 28 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:11:59.992349 20875 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:11:59.992471 20875 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 28 FOLLOWER]: Advancing to term 29
I20251212 03:11:59.993147 20875 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:59.993295 20875 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 29 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:11:59.993573 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 29 candidate_status { last_received { term: 27 index: 54 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:11:59.993592 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 29 candidate_status { last_received { term: 27 index: 54 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:11:59.993664 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 28 FOLLOWER]: Advancing to term 29
I20251212 03:11:59.994318 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 29.
I20251212 03:11:59.994521 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:11:59.994619 20875 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 29 FOLLOWER]: Leader election won for term 29
I20251212 03:11:59.994675 20875 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 29 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:11:59.994743 20875 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 27.54, Last appended by leader: 54, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:11:59.995298 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 28 to 29, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 29 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:00.015465 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 29 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 28. Current term is 29. Ops: []
I20251212 03:12:00.015827 20869 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20251212 03:12:00.015971 20869 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 28 LEADER]: Stepping down as leader of term 28
I20251212 03:12:00.016005 20869 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:00.016077 20869 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 28.55, Last appended by leader: 55, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:00.016148 20869 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 28 FOLLOWER]: Advancing to term 29
I20251212 03:12:00.086858 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 29 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 27 index: 54. Preceding OpId from leader: term: 29 index: 55. (index mismatch)
I20251212 03:12:00.087208 20875 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20251212 03:12:00.088305 20606 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 29 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 28 index: 55. Preceding OpId from leader: term: 29 index: 55. (term mismatch)
I20251212 03:12:00.088367 20606 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting all ops after (but not including) 54
I20251212 03:12:00.088387 20606 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting uncommitted NO_OP operation due to leader change: 28.55
I20251212 03:12:00.088411 20606 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:00.088660 20877 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20251212 03:12:00.188688 20866 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 29 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:00.188812 20866 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 29 FOLLOWER]: Advancing to term 30
I20251212 03:12:00.189440 20866 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:00.189610 20866 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 30 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:00.189881 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 30 candidate_status { last_received { term: 29 index: 55 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:00.189881 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 30 candidate_status { last_received { term: 29 index: 55 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:00.190081 20606 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 29 FOLLOWER]: Advancing to term 30
I20251212 03:12:00.190771 20606 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 30.
I20251212 03:12:00.190871 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 30 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 29. Current term is 30. Ops: []
I20251212 03:12:00.191030 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:00.191123 20870 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 29.55, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20251212 03:12:00.191123 20866 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 30 FOLLOWER]: Leader election won for term 30
I20251212 03:12:00.191260 20866 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 30 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:00.191263 20870 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 29 LEADER]: Stepping down as leader of term 29
I20251212 03:12:00.191330 20870 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:00.191326 20866 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 29.55, Last appended by leader: 55, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:00.191534 20870 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 29.55, Last appended by leader: 55, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:00.191613 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 30 LEADER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 29. Current term is 30. Ops: []
I20251212 03:12:00.191649 20870 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 29 FOLLOWER]: Advancing to term 30
I20251212 03:12:00.191977 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 29 to 30, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 30 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:00.278525 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 30 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 29 index: 55. Preceding OpId from leader: term: 30 index: 56. (index mismatch)
I20251212 03:12:00.278836 20866 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20251212 03:12:00.279867 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 30 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 29 index: 55. Preceding OpId from leader: term: 30 index: 56. (index mismatch)
I20251212 03:12:00.280054 20866 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20251212 03:12:00.380910 20869 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 30 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:00.381019 20869 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 30 FOLLOWER]: Advancing to term 31
I20251212 03:12:00.381667 20869 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:00.381837 20869 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 31 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:00.382090 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 31 candidate_status { last_received { term: 30 index: 56 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:00.382087 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 31 candidate_status { last_received { term: 30 index: 56 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:00.382181 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 30 FOLLOWER]: Advancing to term 31
I20251212 03:12:00.382845 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 31.
I20251212 03:12:00.383092 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:00.383208 20869 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 31 FOLLOWER]: Leader election won for term 31
I20251212 03:12:00.383282 20869 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 31 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:00.383339 20869 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 30.56, Last appended by leader: 56, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:00.383988 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 30 to 31, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 31 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:00.386495 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 31 LEADER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 30. Current term is 31. Ops: []
I20251212 03:12:00.386711 20866 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 30.56, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20251212 03:12:00.386797 20866 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer e9d27948dced4eba8f8228ddc4529393 log is divergent from this leader: its last log entry 31.57 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 56
I20251212 03:12:00.386823 20873 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 30 LEADER]: Stepping down as leader of term 30
I20251212 03:12:00.386873 20873 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:00.386938 20873 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 30.56, Last appended by leader: 56, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:00.387022 20873 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 30 FOLLOWER]: Advancing to term 31
I20251212 03:12:00.471493 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 31 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 30 index: 56. Preceding OpId from leader: term: 31 index: 57. (index mismatch)
I20251212 03:12:00.471858 20869 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20251212 03:12:00.472923 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 31 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 30 index: 56. Preceding OpId from leader: term: 31 index: 57. (index mismatch)
I20251212 03:12:00.473171 20878 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20251212 03:12:00.494483 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:00.573763 20875 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 31 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:00.573865 20875 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 31 FOLLOWER]: Advancing to term 32
I20251212 03:12:00.574605 20875 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:00.574770 20875 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 32 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:00.574981 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 32 candidate_status { last_received { term: 31 index: 57 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:00.575039 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 31 FOLLOWER]: Advancing to term 32
I20251212 03:12:00.574998 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 32 candidate_status { last_received { term: 31 index: 57 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:00.575697 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 32.
I20251212 03:12:00.575904 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:00.575992 20875 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 32 FOLLOWER]: Leader election won for term 32
I20251212 03:12:00.576042 20875 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 32 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:00.576097 20875 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 31.57, Last appended by leader: 57, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:00.576596 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 31 to 32, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 32 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:00.580389 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 32 LEADER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 31. Current term is 32. Ops: []
I20251212 03:12:00.580593 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 32 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 31. Current term is 32. Ops: []
I20251212 03:12:00.580622 20869 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 31.57, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20251212 03:12:00.580699 20869 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer f894b0d567ab4a2ebdcb416d04ef48e5 log is divergent from this leader: its last log entry 32.58 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 57
I20251212 03:12:00.580736 20872 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 31 LEADER]: Stepping down as leader of term 31
I20251212 03:12:00.580775 20872 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:00.580837 20872 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 31.57, Last appended by leader: 57, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:00.580899 20869 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 31.57, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20251212 03:12:00.581032 20872 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 31 FOLLOWER]: Advancing to term 32
I20251212 03:12:00.679858 20873 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:00.679967 20873 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 32 FOLLOWER]: Advancing to term 33
I20251212 03:12:00.680677 20873 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:00.680836 20873 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 33 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:00.681094 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 33 candidate_status { last_received { term: 31 index: 57 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:00.681094 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 33 candidate_status { last_received { term: 31 index: 57 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:00.681226 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 32 FOLLOWER]: Advancing to term 33
I20251212 03:12:00.681879 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 33.
I20251212 03:12:00.682091 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:00.682220 20873 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 33 FOLLOWER]: Leader election won for term 33
I20251212 03:12:00.682278 20873 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 33 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:00.682333 20873 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 31.57, Last appended by leader: 57, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:00.682791 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 32 to 33, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 33 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:00.687038 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 33 LEADER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 32. Current term is 33. Ops: []
I20251212 03:12:00.687322 20875 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20251212 03:12:00.687407 20875 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer d3fe965dfda94fd4bf3f61a5c0452d63 log is divergent from this leader: its last log entry 33.58 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 57
I20251212 03:12:00.687443 20870 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 32 LEADER]: Stepping down as leader of term 32
I20251212 03:12:00.687503 20870 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:00.687575 20870 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 32.58, Last appended by leader: 58, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:00.687656 20870 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 32 FOLLOWER]: Advancing to term 33
I20251212 03:12:00.758447 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 33 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 31 index: 57. Preceding OpId from leader: term: 33 index: 58. (index mismatch)
I20251212 03:12:00.758760 20873 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20251212 03:12:00.759876 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 33 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 32 index: 58. Preceding OpId from leader: term: 33 index: 58. (term mismatch)
I20251212 03:12:00.759931 20759 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting all ops after (but not including) 57
I20251212 03:12:00.759950 20759 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting uncommitted NO_OP operation due to leader change: 32.58
I20251212 03:12:00.759970 20759 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:00.760133 20879 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20251212 03:12:00.860332 20872 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 33 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:00.860476 20872 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 33 FOLLOWER]: Advancing to term 34
I20251212 03:12:00.861166 20872 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:00.861330 20872 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 34 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:00.861640 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 34 candidate_status { last_received { term: 33 index: 58 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:00.861634 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 34 candidate_status { last_received { term: 33 index: 58 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:00.861760 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 33 FOLLOWER]: Advancing to term 34
I20251212 03:12:00.862365 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 34.
I20251212 03:12:00.862654 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:00.862782 20872 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 34 FOLLOWER]: Leader election won for term 34
I20251212 03:12:00.862838 20872 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 34 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:00.862900 20872 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 33.58, Last appended by leader: 58, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:00.863371 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 33 to 34, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 34 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:00.875758 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 34 LEADER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 33. Current term is 34. Ops: []
I20251212 03:12:00.876031 20879 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 33.58, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20251212 03:12:00.876145 20879 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer e9d27948dced4eba8f8228ddc4529393 log is divergent from this leader: its last log entry 34.59 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 58
I20251212 03:12:00.876189 20873 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 33 LEADER]: Stepping down as leader of term 33
I20251212 03:12:00.876242 20873 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:00.876299 20873 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 33.58, Last appended by leader: 58, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:00.876395 20873 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 33 FOLLOWER]: Advancing to term 34
I20251212 03:12:00.946931 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 34 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 33 index: 58. Preceding OpId from leader: term: 34 index: 59. (index mismatch)
I20251212 03:12:00.947206 20872 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20251212 03:12:00.948437 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 34 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 33 index: 58. Preceding OpId from leader: term: 34 index: 59. (index mismatch)
I20251212 03:12:00.948714 20878 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20251212 03:12:01.131918 20870 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 34 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:01.132028 20870 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 34 FOLLOWER]: Advancing to term 35
I20251212 03:12:01.132690 20870 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:01.132872 20870 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 35 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:01.133061 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 35 candidate_status { last_received { term: 34 index: 59 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:01.133127 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 35 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 34. Current term is 35. Ops: []
I20251212 03:12:01.133067 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 35 candidate_status { last_received { term: 34 index: 59 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:01.133348 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:01.133371 20869 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 34.59, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20251212 03:12:01.133443 20870 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20251212 03:12:01.133514 20872 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 34 LEADER]: Stepping down as leader of term 34
I20251212 03:12:01.133551 20872 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:01.133610 20872 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 34.59, Last appended by leader: 59, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:01.133680 20872 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 34 FOLLOWER]: Advancing to term 35
I20251212 03:12:01.143990 20879 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 34 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:01.144075 20879 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 34 FOLLOWER]: Advancing to term 35
I20251212 03:12:01.144754 20879 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:01.144927 20879 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 35 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:01.145148 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 35 candidate_status { last_received { term: 34 index: 59 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:01.145145 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 35 candidate_status { last_received { term: 34 index: 59 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:01.145255 20759 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate d3fe965dfda94fd4bf3f61a5c0452d63 in current term 35: Already voted for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in this term.
I20251212 03:12:01.145829 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 35.
I20251212 03:12:01.146013 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:01.146111 20879 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 35 FOLLOWER]: Leader election won for term 35
I20251212 03:12:01.146164 20879 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 35 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:01.146227 20879 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 34.59, Last appended by leader: 59, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:01.146680 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 34 to 35, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 35 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:01.234295 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 35 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 34 index: 59. Preceding OpId from leader: term: 35 index: 60. (index mismatch)
I20251212 03:12:01.234642 20879 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20251212 03:12:01.235703 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 35 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 34 index: 59. Preceding OpId from leader: term: 35 index: 60. (index mismatch)
I20251212 03:12:01.235924 20879 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20251212 03:12:01.336436 20870 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 35 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:01.336541 20870 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 35 FOLLOWER]: Advancing to term 36
I20251212 03:12:01.337204 20870 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:01.337373 20870 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 36 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:01.337597 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 36 candidate_status { last_received { term: 35 index: 60 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:01.337597 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 36 candidate_status { last_received { term: 35 index: 60 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:01.337836 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:01.337919 20870 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20251212 03:12:01.349347 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 36 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 35. Current term is 36. Ops: []
I20251212 03:12:01.349660 20880 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 35.60, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20251212 03:12:01.349782 20880 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 35 LEADER]: Stepping down as leader of term 35
I20251212 03:12:01.349813 20880 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:01.349874 20880 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 35.60, Last appended by leader: 60, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:01.349951 20880 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 35 FOLLOWER]: Advancing to term 36
I20251212 03:12:01.426405 20872 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 35 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:01.426525 20872 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 35 FOLLOWER]: Advancing to term 36
I20251212 03:12:01.427184 20872 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:01.427343 20872 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 36 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:01.427606 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 36 candidate_status { last_received { term: 35 index: 60 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:01.427626 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 36 candidate_status { last_received { term: 35 index: 60 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:01.427731 20759 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 in current term 36: Already voted for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in this term.
I20251212 03:12:01.428303 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 36.
I20251212 03:12:01.428527 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:01.428639 20872 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 36 FOLLOWER]: Leader election won for term 36
I20251212 03:12:01.428730 20872 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 36 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:01.428799 20872 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 35.60, Last appended by leader: 60, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:01.429270 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 35 to 36, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 36 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:01.443337 20870 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:01.443485 20870 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 36 FOLLOWER]: Advancing to term 37
I20251212 03:12:01.444164 20870 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:01.444336 20870 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 37 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:01.444578 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 37 candidate_status { last_received { term: 35 index: 60 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:01.444590 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 37 candidate_status { last_received { term: 35 index: 60 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:01.444672 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 36 FOLLOWER]: Advancing to term 37
I20251212 03:12:01.445230 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 37.
I20251212 03:12:01.445400 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:01.445516 20870 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 37 FOLLOWER]: Leader election won for term 37
I20251212 03:12:01.445636 20870 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 37 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:01.445698 20870 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 35.60, Last appended by leader: 60, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:01.446198 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 36 to 37, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 37 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:01.494810 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:01.524935 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 37 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 35 index: 60. Preceding OpId from leader: term: 37 index: 61. (index mismatch)
I20251212 03:12:01.525245 20870 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20251212 03:12:01.526539 20605 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 36 LEADER]: Stepping down as leader of term 36
I20251212 03:12:01.526603 20605 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:01.526659 20605 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 36.61, Last appended by leader: 61, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:01.526752 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 36 FOLLOWER]: Advancing to term 37
I20251212 03:12:01.527449 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 37 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 36 index: 61. Preceding OpId from leader: term: 37 index: 61. (term mismatch)
I20251212 03:12:01.527482 20605 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting all ops after (but not including) 60
I20251212 03:12:01.527504 20605 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting uncommitted NO_OP operation due to leader change: 36.61
I20251212 03:12:01.527518 20605 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:01.527727 20870 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20251212 03:12:01.626734 20880 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 37 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:01.626832 20880 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 37 FOLLOWER]: Advancing to term 38
I20251212 03:12:01.627506 20880 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:01.627667 20880 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 38 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:01.627851 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 38 candidate_status { last_received { term: 37 index: 61 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:01.627857 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 38 candidate_status { last_received { term: 37 index: 61 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:01.628110 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:01.628203 20880 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20251212 03:12:01.628301 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 38 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 37. Current term is 38. Ops: []
I20251212 03:12:01.628513 20881 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 37.61, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20251212 03:12:01.628595 20881 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 37 LEADER]: Stepping down as leader of term 37
I20251212 03:12:01.628614 20881 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:01.628676 20881 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 37.61, Last appended by leader: 61, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:01.628754 20881 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 37 FOLLOWER]: Advancing to term 38
I20251212 03:12:01.752616 20869 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 37 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:01.752722 20869 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 37 FOLLOWER]: Advancing to term 38
I20251212 03:12:01.754302 20869 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:01.754458 20869 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 38 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:01.754706 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 38 candidate_status { last_received { term: 37 index: 61 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:01.754721 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 38 candidate_status { last_received { term: 37 index: 61 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:01.754865 20682 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 in current term 38: Already voted for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in this term.
I20251212 03:12:01.755321 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 38.
I20251212 03:12:01.755501 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:01.755609 20869 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 38 FOLLOWER]: Leader election won for term 38
I20251212 03:12:01.755673 20869 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 38 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:01.755730 20869 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 37.61, Last appended by leader: 61, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:01.756186 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 37 to 38, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 38 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:01.760637 20880 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:01.760704 20880 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 38 FOLLOWER]: Advancing to term 39
I20251212 03:12:01.761246 20880 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:01.761370 20880 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 39 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:01.761544 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 39 candidate_status { last_received { term: 37 index: 61 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:01.761540 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 39 candidate_status { last_received { term: 37 index: 61 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:01.761655 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 38 FOLLOWER]: Advancing to term 39
I20251212 03:12:01.762167 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 39.
I20251212 03:12:01.762398 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:01.762524 20880 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 39 FOLLOWER]: Leader election won for term 39
I20251212 03:12:01.762578 20880 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 39 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:01.762634 20880 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 37.61, Last appended by leader: 61, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:01.763101 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 38 to 39, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 39 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:01.837647 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 39 LEADER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 38. Current term is 39. Ops: []
I20251212 03:12:01.837978 20869 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20251212 03:12:01.838085 20869 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer d3fe965dfda94fd4bf3f61a5c0452d63 log is divergent from this leader: its last log entry 39.62 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 61
I20251212 03:12:01.838127 20872 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 38 LEADER]: Stepping down as leader of term 38
I20251212 03:12:01.838182 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 39 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 37 index: 61. Preceding OpId from leader: term: 39 index: 62. (index mismatch)
I20251212 03:12:01.838222 20872 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:01.838305 20872 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 38.62, Last appended by leader: 62, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:01.838407 20872 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 38 FOLLOWER]: Advancing to term 39
I20251212 03:12:01.838402 20880 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20251212 03:12:01.839505 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 39 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 38 index: 62. Preceding OpId from leader: term: 39 index: 62. (term mismatch)
I20251212 03:12:01.839570 20605 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting all ops after (but not including) 61
I20251212 03:12:01.839592 20605 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting uncommitted NO_OP operation due to leader change: 38.62
I20251212 03:12:01.839617 20605 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 39 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:01.839788 20883 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20251212 03:12:01.939781 20881 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 39 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:01.939875 20881 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 39 FOLLOWER]: Advancing to term 40
I20251212 03:12:01.940570 20881 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:01.940737 20881 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 40 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:01.940945 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 40 candidate_status { last_received { term: 39 index: 62 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:01.940945 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 40 candidate_status { last_received { term: 39 index: 62 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:01.941201 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:01.941308 20881 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20251212 03:12:01.944586 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 40 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 39. Current term is 40. Ops: []
I20251212 03:12:01.944834 20880 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 39.62, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20251212 03:12:01.944967 20880 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 39 LEADER]: Stepping down as leader of term 39
I20251212 03:12:01.944998 20880 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:01.945062 20880 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 39.62, Last appended by leader: 62, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:01.945144 20880 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 39 FOLLOWER]: Advancing to term 40
I20251212 03:12:02.063934 20872 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 39 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:02.064047 20872 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 39 FOLLOWER]: Advancing to term 40
I20251212 03:12:02.064790 20872 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:02.064955 20872 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 40 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:02.065210 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 40 candidate_status { last_received { term: 39 index: 62 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:02.065228 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 40 candidate_status { last_received { term: 39 index: 62 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:02.065320 20759 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 in current term 40: Already voted for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in this term.
I20251212 03:12:02.065860 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 40.
I20251212 03:12:02.066053 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:02.066184 20872 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 40 FOLLOWER]: Leader election won for term 40
I20251212 03:12:02.066296 20872 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 40 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:02.066365 20872 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 39.62, Last appended by leader: 62, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:02.066983 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 39 to 40, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 40 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:02.068714 20881 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:02.068771 20881 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 40 FOLLOWER]: Advancing to term 41
I20251212 03:12:02.069322 20881 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:02.069459 20881 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 41 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:02.069633 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 41 candidate_status { last_received { term: 39 index: 62 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:02.069665 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 41 candidate_status { last_received { term: 39 index: 62 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:02.069708 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 40 FOLLOWER]: Advancing to term 41
I20251212 03:12:02.070375 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 41.
I20251212 03:12:02.070609 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:02.070712 20881 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 41 FOLLOWER]: Leader election won for term 41
I20251212 03:12:02.070771 20881 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 41 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:02.070822 20881 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 39.62, Last appended by leader: 62, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:02.071347 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 40 to 41, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 41 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:02.144424 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 41 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 40. Current term is 41. Ops: []
I20251212 03:12:02.144826 20872 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20251212 03:12:02.144958 20872 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 40 LEADER]: Stepping down as leader of term 40
I20251212 03:12:02.145002 20872 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:02.145071 20872 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 40.63, Last appended by leader: 63, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:02.145144 20872 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 40 FOLLOWER]: Advancing to term 41
I20251212 03:12:02.152768 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 41 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 39 index: 62. Preceding OpId from leader: term: 41 index: 63. (index mismatch)
I20251212 03:12:02.153031 20881 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20251212 03:12:02.154296 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 41 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 40 index: 63. Preceding OpId from leader: term: 41 index: 63. (term mismatch)
I20251212 03:12:02.154356 20605 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting all ops after (but not including) 62
I20251212 03:12:02.154376 20605 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting uncommitted NO_OP operation due to leader change: 40.63
I20251212 03:12:02.154402 20605 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 41 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:02.154628 20870 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20251212 03:12:02.254913 20880 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 41 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:02.255015 20880 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 41 FOLLOWER]: Advancing to term 42
I20251212 03:12:02.255717 20880 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:02.255900 20880 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 42 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:02.256156 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 42 candidate_status { last_received { term: 41 index: 63 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:02.256156 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 42 candidate_status { last_received { term: 41 index: 63 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:02.256489 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:02.256583 20880 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20251212 03:12:02.258564 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 42 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 41. Current term is 42. Ops: []
I20251212 03:12:02.258777 20881 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 41.63, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20251212 03:12:02.258877 20881 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 41 LEADER]: Stepping down as leader of term 41
I20251212 03:12:02.258908 20881 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:02.258965 20881 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 41.63, Last appended by leader: 63, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:02.259037 20881 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 41 FOLLOWER]: Advancing to term 42
I20251212 03:12:02.361066 20880 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:02.361173 20880 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 42 FOLLOWER]: Advancing to term 43
I20251212 03:12:02.361904 20880 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:02.362174 20880 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 43 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:02.362332 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 43 candidate_status { last_received { term: 41 index: 63 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:02.362304 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 43 candidate_status { last_received { term: 41 index: 63 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:02.362419 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 42 FOLLOWER]: Advancing to term 43
I20251212 03:12:02.362427 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 41 FOLLOWER]: Advancing to term 43
I20251212 03:12:02.363202 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 43.
I20251212 03:12:02.363202 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 43.
I20251212 03:12:02.363494 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters:
I20251212 03:12:02.363618 20880 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 43 FOLLOWER]: Leader election won for term 43
I20251212 03:12:02.363715 20880 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 43 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:02.363782 20880 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 41.63, Last appended by leader: 63, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:02.364562 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 41 to 43, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 43 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:02.454838 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 43 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 41 index: 63. Preceding OpId from leader: term: 43 index: 64. (index mismatch)
I20251212 03:12:02.455119 20880 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20251212 03:12:02.456182 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 43 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 41 index: 63. Preceding OpId from leader: term: 43 index: 64. (index mismatch)
I20251212 03:12:02.456444 20885 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20251212 03:12:02.495119 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:02.732344 20872 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 43 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:02.732451 20872 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 43 FOLLOWER]: Advancing to term 44
I20251212 03:12:02.733142 20872 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:02.733314 20872 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 44 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:02.733597 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 44 candidate_status { last_received { term: 43 index: 64 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:02.733597 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 44 candidate_status { last_received { term: 43 index: 64 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:02.733920 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 44 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:02.734016 20872 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 44 FOLLOWER]: Leader election lost for term 44. Reason: could not achieve majority
I20251212 03:12:02.745338 20881 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 43 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:02.745430 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 44 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 43. Current term is 44. Ops: []
I20251212 03:12:02.745467 20881 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 43 FOLLOWER]: Advancing to term 44
I20251212 03:12:02.745713 20880 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 43.64, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20251212 03:12:02.745827 20880 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 43 LEADER]: Stepping down as leader of term 43
I20251212 03:12:02.745851 20880 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:02.745903 20880 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 43.64, Last appended by leader: 64, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:02.745975 20880 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 43 FOLLOWER]: Advancing to term 44
I20251212 03:12:02.746394 20881 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:02.746632 20881 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 44 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:02.746721 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 44 candidate_status { last_received { term: 43 index: 64 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:02.746743 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 44 candidate_status { last_received { term: 43 index: 64 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:02.746809 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 44: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:02.747452 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 44.
I20251212 03:12:02.747625 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:02.747721 20881 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 44 FOLLOWER]: Leader election won for term 44
I20251212 03:12:02.747776 20881 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 44 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:02.747813 20881 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 43.64, Last appended by leader: 64, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:02.748271 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 43 to 44, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 44 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:02.844799 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 44 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 43 index: 64. Preceding OpId from leader: term: 44 index: 65. (index mismatch)
I20251212 03:12:02.845152 20881 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20251212 03:12:02.846288 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 44 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 43 index: 64. Preceding OpId from leader: term: 44 index: 65. (index mismatch)
I20251212 03:12:02.846525 20870 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20251212 03:12:03.023996 20872 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 44 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:03.024106 20872 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 44 FOLLOWER]: Advancing to term 45
I20251212 03:12:03.024816 20872 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:03.024981 20872 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 45 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:03.025216 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 45 candidate_status { last_received { term: 44 index: 65 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:03.025245 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 45 candidate_status { last_received { term: 44 index: 65 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:03.025271 20880 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 44 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:03.025372 20880 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 44 FOLLOWER]: Advancing to term 45
I20251212 03:12:03.025952 20880 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:03.026088 20682 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 in current term 45: Already voted for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in this term.
I20251212 03:12:03.026196 20880 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 45 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:03.026250 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 45 candidate_status { last_received { term: 44 index: 65 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:03.026309 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:03.026338 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 45 candidate_status { last_received { term: 44 index: 65 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:03.026388 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate d3fe965dfda94fd4bf3f61a5c0452d63 in current term 45: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:03.026427 20872 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
I20251212 03:12:03.026505 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:03.026578 20880 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
I20251212 03:12:03.028770 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 45 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 44. Current term is 45. Ops: []
I20251212 03:12:03.029011 20881 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 44.65, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20251212 03:12:03.029120 20881 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 44 LEADER]: Stepping down as leader of term 44
I20251212 03:12:03.029160 20881 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 44 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:03.029208 20881 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 44.65, Last appended by leader: 65, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:03.029286 20881 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 44 FOLLOWER]: Advancing to term 45
I20251212 03:12:03.148000 20880 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:03.148108 20880 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 45 FOLLOWER]: Advancing to term 46
I20251212 03:12:03.148797 20880 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:03.148959 20880 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 46 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:03.149219 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 46 candidate_status { last_received { term: 44 index: 65 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:03.149226 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 46 candidate_status { last_received { term: 44 index: 65 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:03.149291 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 45 FOLLOWER]: Advancing to term 46
I20251212 03:12:03.149355 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 45 FOLLOWER]: Advancing to term 46
I20251212 03:12:03.149945 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 46.
I20251212 03:12:03.149950 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 46.
I20251212 03:12:03.150137 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters:
I20251212 03:12:03.150281 20880 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 46 FOLLOWER]: Leader election won for term 46
I20251212 03:12:03.150344 20880 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 46 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:03.150400 20880 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 44.65, Last appended by leader: 65, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:03.150883 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 44 to 46, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 46 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:03.237573 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 46 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 44 index: 65. Preceding OpId from leader: term: 46 index: 66. (index mismatch)
I20251212 03:12:03.237888 20880 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20251212 03:12:03.239035 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 46 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 44 index: 65. Preceding OpId from leader: term: 46 index: 66. (index mismatch)
I20251212 03:12:03.239256 20880 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20251212 03:12:03.427796 20881 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 46 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:03.427891 20881 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 46 FOLLOWER]: Advancing to term 47
I20251212 03:12:03.428524 20881 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:03.428680 20881 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 47 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:03.428913 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 47 candidate_status { last_received { term: 46 index: 66 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:03.428913 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 47 candidate_status { last_received { term: 46 index: 66 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:03.429162 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:03.429311 20881 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 47 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
I20251212 03:12:03.447250 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 47 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 46. Current term is 47. Ops: []
I20251212 03:12:03.447588 20885 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 46.66, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20251212 03:12:03.447728 20885 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 46 LEADER]: Stepping down as leader of term 46
I20251212 03:12:03.447772 20885 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 46 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:03.447835 20885 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 46.66, Last appended by leader: 66, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:03.447916 20885 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 46 FOLLOWER]: Advancing to term 47
I20251212 03:12:03.495368 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:03.522003 20886 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 46 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:03.522109 20886 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 46 FOLLOWER]: Advancing to term 47
I20251212 03:12:03.522907 20886 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:03.523082 20886 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 47 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:03.523319 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 47 candidate_status { last_received { term: 46 index: 66 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:03.523396 20759 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 in current term 47: Already voted for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in this term.
I20251212 03:12:03.523332 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 47 candidate_status { last_received { term: 46 index: 66 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:03.523995 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 47.
I20251212 03:12:03.524194 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:03.524286 20886 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 47 FOLLOWER]: Leader election won for term 47
I20251212 03:12:03.524354 20886 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 47 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:03.524412 20886 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 46.66, Last appended by leader: 66, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:03.524932 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 46 to 47, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 47 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:03.573724 20881 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:03.573846 20881 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 47 FOLLOWER]: Advancing to term 48
I20251212 03:12:03.574632 20881 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:03.574821 20881 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 48 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:03.575109 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 48 candidate_status { last_received { term: 46 index: 66 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:03.575109 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 48 candidate_status { last_received { term: 46 index: 66 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:03.575251 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 47 FOLLOWER]: Advancing to term 48
I20251212 03:12:03.575876 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 48.
I20251212 03:12:03.576088 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:03.576196 20881 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 48 FOLLOWER]: Leader election won for term 48
I20251212 03:12:03.576297 20881 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 48 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:03.576341 20881 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 46.66, Last appended by leader: 66, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:03.576809 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 47 to 48, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 48 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:03.624436 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 48 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 47. Current term is 48. Ops: []
I20251212 03:12:03.624796 20886 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20251212 03:12:03.624902 20886 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 47 LEADER]: Stepping down as leader of term 47
I20251212 03:12:03.624922 20886 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 47 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:03.624989 20886 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 47.67, Last appended by leader: 67, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:03.625066 20886 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 47 FOLLOWER]: Advancing to term 48
I20251212 03:12:03.683359 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 48 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 47 index: 67. Preceding OpId from leader: term: 48 index: 67. (term mismatch)
I20251212 03:12:03.683465 20605 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting all ops after (but not including) 66
I20251212 03:12:03.683490 20605 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting uncommitted NO_OP operation due to leader change: 47.67
I20251212 03:12:03.683511 20605 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 48 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:03.683882 20881 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20251212 03:12:03.684091 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 48 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 46 index: 66. Preceding OpId from leader: term: 48 index: 67. (index mismatch)
I20251212 03:12:03.684401 20881 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20251212 03:12:03.785954 20885 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 48 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:03.786056 20885 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 48 FOLLOWER]: Advancing to term 49
I20251212 03:12:03.786774 20885 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:03.787045 20885 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 49 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:03.787174 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 49 candidate_status { last_received { term: 48 index: 67 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:03.787238 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 48 FOLLOWER]: Advancing to term 49
I20251212 03:12:03.787223 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 49 candidate_status { last_received { term: 48 index: 67 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:03.787957 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 49.
I20251212 03:12:03.788165 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:03.788254 20885 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 49 FOLLOWER]: Leader election won for term 49
I20251212 03:12:03.788305 20885 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 49 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:03.788362 20885 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 48.67, Last appended by leader: 67, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:03.788842 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 48 to 49, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 49 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:03.795969 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 49 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 48. Current term is 49. Ops: []
I20251212 03:12:03.796249 20887 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 48.67, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20251212 03:12:03.796382 20881 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 48 LEADER]: Stepping down as leader of term 48
I20251212 03:12:03.796478 20881 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 48 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:03.796540 20881 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 48.67, Last appended by leader: 67, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:03.796633 20881 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 48 FOLLOWER]: Advancing to term 49
I20251212 03:12:03.903851 20886 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 49 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:03.903949 20886 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 49 FOLLOWER]: Advancing to term 50
I20251212 03:12:03.904600 20886 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:03.904775 20886 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 50 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:03.905007 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 50 candidate_status { last_received { term: 48 index: 67 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:03.905022 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 50 candidate_status { last_received { term: 48 index: 67 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:03.905102 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 49 FOLLOWER]: Advancing to term 50
I20251212 03:12:03.905676 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 50.
I20251212 03:12:03.905884 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:03.906010 20886 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 50 FOLLOWER]: Leader election won for term 50
I20251212 03:12:03.906131 20886 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 50 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:03.906183 20886 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 48.67, Last appended by leader: 67, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:03.906518 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 50 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 49. Current term is 50. Ops: []
I20251212 03:12:03.906724 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 49 to 50, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 50 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:03.906734 20885 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20251212 03:12:03.906868 20885 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 49 LEADER]: Stepping down as leader of term 49
I20251212 03:12:03.906895 20885 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 49 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:03.906942 20885 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 49.68, Last appended by leader: 68, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:03.907004 20885 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 49 FOLLOWER]: Advancing to term 50
I20251212 03:12:03.986305 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 50 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 49 index: 68. Preceding OpId from leader: term: 50 index: 68. (term mismatch)
I20251212 03:12:03.986392 20682 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting all ops after (but not including) 67
I20251212 03:12:03.986436 20682 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting uncommitted NO_OP operation due to leader change: 49.68
I20251212 03:12:03.986451 20682 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 50 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:03.986788 20886 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20251212 03:12:03.988015 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 50 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 48 index: 67. Preceding OpId from leader: term: 50 index: 68. (index mismatch)
I20251212 03:12:03.988266 20888 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20251212 03:12:04.089694 20887 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 50 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:04.089819 20887 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 50 FOLLOWER]: Advancing to term 51
I20251212 03:12:04.090678 20887 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:04.090878 20887 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 51 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:04.091126 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 51 candidate_status { last_received { term: 50 index: 68 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:04.091132 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 51 candidate_status { last_received { term: 50 index: 68 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:04.091320 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 50 FOLLOWER]: Advancing to term 51
I20251212 03:12:04.091316 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 51 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 50. Current term is 51. Ops: []
I20251212 03:12:04.091797 20886 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 50.68, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20251212 03:12:04.091925 20872 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 50 LEADER]: Stepping down as leader of term 50
I20251212 03:12:04.091967 20872 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 50 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:04.092017 20872 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 50.68, Last appended by leader: 68, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:04.092103 20872 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 50 FOLLOWER]: Advancing to term 51
I20251212 03:12:04.092427 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 51.
I20251212 03:12:04.092628 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:04.092787 20887 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 51 FOLLOWER]: Leader election won for term 51
I20251212 03:12:04.092864 20887 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 51 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:04.092927 20887 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 50.68, Last appended by leader: 68, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:04.093422 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 50 to 51, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 51 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:04.196945 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 51 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 50 index: 68. Preceding OpId from leader: term: 51 index: 69. (index mismatch)
I20251212 03:12:04.197288 20887 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20251212 03:12:04.198453 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 51 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 50 index: 68. Preceding OpId from leader: term: 51 index: 69. (index mismatch)
I20251212 03:12:04.198685 20881 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20251212 03:12:04.299254 20885 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 51 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:04.299356 20885 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 51 FOLLOWER]: Advancing to term 52
I20251212 03:12:04.300130 20885 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:04.300285 20885 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 52 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:04.300557 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 52 candidate_status { last_received { term: 51 index: 69 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:04.300628 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 51 FOLLOWER]: Advancing to term 52
I20251212 03:12:04.300562 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 52 candidate_status { last_received { term: 51 index: 69 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:04.301177 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 52.
I20251212 03:12:04.301473 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:04.301599 20885 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 52 FOLLOWER]: Leader election won for term 52
I20251212 03:12:04.301677 20885 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 52 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:04.301738 20885 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 51.69, Last appended by leader: 69, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:04.302191 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 52 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 51. Current term is 52. Ops: []
I20251212 03:12:04.302222 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 51 to 52, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 52 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:04.302455 20887 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 51.69, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20251212 03:12:04.302548 20887 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 51 LEADER]: Stepping down as leader of term 51
I20251212 03:12:04.302579 20887 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 51 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:04.302628 20887 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 51.69, Last appended by leader: 69, Current term: 51, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:04.302820 20887 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 51 FOLLOWER]: Advancing to term 52
I20251212 03:12:04.398741 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 52 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 51 index: 69. Preceding OpId from leader: term: 52 index: 70. (index mismatch)
I20251212 03:12:04.399037 20885 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20251212 03:12:04.400180 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 52 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 51 index: 69. Preceding OpId from leader: term: 52 index: 70. (index mismatch)
I20251212 03:12:04.400352 20880 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20251212 03:12:04.495604 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:04.501277 20872 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 52 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:04.501361 20872 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 52 FOLLOWER]: Advancing to term 53
I20251212 03:12:04.502096 20872 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:04.502297 20872 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 53 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:04.502482 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 53 candidate_status { last_received { term: 52 index: 70 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:04.502496 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 53 candidate_status { last_received { term: 52 index: 70 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:04.502583 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 52 FOLLOWER]: Advancing to term 53
I20251212 03:12:04.503181 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 53.
I20251212 03:12:04.503362 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:04.503496 20872 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 53 FOLLOWER]: Leader election won for term 53
I20251212 03:12:04.503569 20872 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 53 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:04.503626 20872 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 52.70, Last appended by leader: 70, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:04.504141 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 52 to 53, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 53 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:04.513170 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 53 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 52. Current term is 53. Ops: []
I20251212 03:12:04.513464 20880 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 52.70, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20251212 03:12:04.513572 20880 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 52 LEADER]: Stepping down as leader of term 52
I20251212 03:12:04.513604 20880 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 52 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:04.513659 20880 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 52.70, Last appended by leader: 70, Current term: 52, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:04.513733 20880 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 52 FOLLOWER]: Advancing to term 53
I20251212 03:12:04.591718 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 53 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 52 index: 70. Preceding OpId from leader: term: 53 index: 71. (index mismatch)
I20251212 03:12:04.591997 20872 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20251212 03:12:04.593001 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 53 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 52 index: 70. Preceding OpId from leader: term: 53 index: 71. (index mismatch)
I20251212 03:12:04.593276 20872 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20251212 03:12:04.789793 20887 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 53 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:04.789897 20887 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 53 FOLLOWER]: Advancing to term 54
I20251212 03:12:04.790597 20887 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 54 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:04.790753 20887 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 54 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:04.790963 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 54 candidate_status { last_received { term: 53 index: 71 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:04.790977 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 54 candidate_status { last_received { term: 53 index: 71 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:04.791280 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 54 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:04.791392 20887 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 54 FOLLOWER]: Leader election lost for term 54. Reason: could not achieve majority
I20251212 03:12:04.792399 20880 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 53 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:04.792462 20880 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 53 FOLLOWER]: Advancing to term 54
I20251212 03:12:04.793023 20880 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 54 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:04.793139 20880 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 54 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:04.793341 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 54 candidate_status { last_received { term: 53 index: 71 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:04.793377 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 54 candidate_status { last_received { term: 53 index: 71 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:04.793431 20759 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 54 FOLLOWER]: Leader election vote request: Denying vote to candidate d3fe965dfda94fd4bf3f61a5c0452d63 in current term 54: Already voted for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in this term.
I20251212 03:12:04.793581 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 54 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:04.793679 20880 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 54 FOLLOWER]: Leader election lost for term 54. Reason: could not achieve majority
I20251212 03:12:04.802912 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 54 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 53. Current term is 54. Ops: []
I20251212 03:12:04.803161 20891 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 53.71, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20251212 03:12:04.803284 20891 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 53 LEADER]: Stepping down as leader of term 53
I20251212 03:12:04.803303 20891 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 53 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:04.803355 20891 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 53.71, Last appended by leader: 71, Current term: 53, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:04.803434 20891 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 53 FOLLOWER]: Advancing to term 54
I20251212 03:12:04.921144 20880 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 54 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:04.921240 20880 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 54 FOLLOWER]: Advancing to term 55
I20251212 03:12:04.921923 20880 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 55 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:04.922080 20880 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 55 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:04.922312 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 55 candidate_status { last_received { term: 53 index: 71 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:04.922312 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 55 candidate_status { last_received { term: 53 index: 71 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:04.922465 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 54 FOLLOWER]: Advancing to term 55
I20251212 03:12:04.922480 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 54 FOLLOWER]: Advancing to term 55
I20251212 03:12:04.923116 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 55 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 55.
I20251212 03:12:04.923116 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 55 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 55.
I20251212 03:12:04.923338 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 55 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters:
I20251212 03:12:04.923468 20880 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 55 FOLLOWER]: Leader election won for term 55
I20251212 03:12:04.923520 20880 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 55 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:04.923576 20880 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 53.71, Last appended by leader: 71, Current term: 55, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:04.924019 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 53 to 55, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 55 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:05.036582 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 55 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 53 index: 71. Preceding OpId from leader: term: 55 index: 72. (index mismatch)
I20251212 03:12:05.036880 20880 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20251212 03:12:05.038021 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 55 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 53 index: 71. Preceding OpId from leader: term: 55 index: 72. (index mismatch)
I20251212 03:12:05.038225 20891 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 55 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:05.038236 20880 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20251212 03:12:05.038341 20891 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 55 FOLLOWER]: Advancing to term 56
I20251212 03:12:05.038998 20891 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 56 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:05.039125 20891 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 56 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:05.039131 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 56 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 55. Current term is 56. Ops: [55.72-55.72]
I20251212 03:12:05.039253 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 56 candidate_status { last_received { term: 53 index: 71 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:05.039340 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 56 candidate_status { last_received { term: 53 index: 71 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:05.039351 20880 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 53.71, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20251212 03:12:05.039527 20880 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 55 LEADER]: Stepping down as leader of term 55
I20251212 03:12:05.039563 20880 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 55 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:05.039603 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 56 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:05.039613 20880 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 55.72, Last appended by leader: 72, Current term: 55, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:05.039783 20891 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 56 FOLLOWER]: Leader election lost for term 56. Reason: could not achieve majority
I20251212 03:12:05.039834 20880 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 55 FOLLOWER]: Advancing to term 56
I20251212 03:12:05.138576 20887 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 55 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:05.138877 20887 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 55 FOLLOWER]: Advancing to term 56
I20251212 03:12:05.139609 20887 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 56 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:05.139772 20887 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 56 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:05.140029 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 56 candidate_status { last_received { term: 55 index: 72 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:05.140043 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 56 candidate_status { last_received { term: 55 index: 72 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:05.140111 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 56: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:05.140869 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 56 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 56.
I20251212 03:12:05.141100 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 56 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:05.141214 20887 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 56 FOLLOWER]: Leader election won for term 56
I20251212 03:12:05.141323 20887 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 56 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:05.141371 20887 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 55.72, Last appended by leader: 72, Current term: 56, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:05.141875 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 55 to 56, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 56 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:05.145207 20891 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 56 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:05.145278 20891 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 56 FOLLOWER]: Advancing to term 57
I20251212 03:12:05.145957 20891 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 57 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:05.146117 20891 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 57 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:05.146327 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 57 candidate_status { last_received { term: 53 index: 71 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:05.146334 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 57 candidate_status { last_received { term: 53 index: 71 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:05.146538 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 56 FOLLOWER]: Advancing to term 57
I20251212 03:12:05.147058 20682 raft_consensus.cc:2410] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 57 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 for term 57 because replica has last-logged OpId of term: 55 index: 72, which is greater than that of the candidate, which has last-logged OpId of term: 53 index: 71.
I20251212 03:12:05.147258 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 57 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:05.147354 20891 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 57 FOLLOWER]: Leader election lost for term 57. Reason: could not achieve majority
I20251212 03:12:05.264605 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 57 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 56. Current term is 57. Ops: []
I20251212 03:12:05.264950 20887 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20251212 03:12:05.265066 20887 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 56 LEADER]: Stepping down as leader of term 56
I20251212 03:12:05.265098 20887 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 56 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:05.265167 20887 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 56.73, Last appended by leader: 73, Current term: 56, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:05.265229 20880 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 57 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:05.265244 20887 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 56 FOLLOWER]: Advancing to term 57
I20251212 03:12:05.265322 20880 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 57 FOLLOWER]: Advancing to term 58
I20251212 03:12:05.265980 20880 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 58 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:05.266180 20880 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 58 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:05.266297 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 58 candidate_status { last_received { term: 55 index: 72 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:05.266299 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 58 candidate_status { last_received { term: 55 index: 72 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:05.266388 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 57 FOLLOWER]: Advancing to term 58
I20251212 03:12:05.266403 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 57 FOLLOWER]: Advancing to term 58
I20251212 03:12:05.267035 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 58 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 58.
I20251212 03:12:05.267051 20759 raft_consensus.cc:2410] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 58 FOLLOWER]: Leader election vote request: Denying vote to candidate d3fe965dfda94fd4bf3f61a5c0452d63 for term 58 because replica has last-logged OpId of term: 56 index: 73, which is greater than that of the candidate, which has last-logged OpId of term: 55 index: 72.
I20251212 03:12:05.267243 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 58 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters:
I20251212 03:12:05.267379 20880 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 58 FOLLOWER]: Leader election won for term 58
I20251212 03:12:05.267477 20880 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 58 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:05.267532 20880 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 55.72, Last appended by leader: 72, Current term: 58, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:05.267997 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 56 to 58, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 58 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:05.344327 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 58 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 56 index: 73. Preceding OpId from leader: term: 58 index: 73. (term mismatch)
I20251212 03:12:05.344419 20759 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting all ops after (but not including) 72
I20251212 03:12:05.344436 20759 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting uncommitted NO_OP operation due to leader change: 56.73
I20251212 03:12:05.344450 20759 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 58 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:05.344705 20880 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20251212 03:12:05.345752 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 58 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 53 index: 71. Preceding OpId from leader: term: 58 index: 73. (index mismatch)
I20251212 03:12:05.345944 20892 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 71, Time since last communication: 0.000s
I20251212 03:12:05.495834 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:05.523458 20891 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 58 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:05.523558 20891 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 58 FOLLOWER]: Advancing to term 59
I20251212 03:12:05.524206 20891 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 59 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:05.524364 20891 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 59 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:05.524580 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 59 candidate_status { last_received { term: 58 index: 73 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:05.524609 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 59 candidate_status { last_received { term: 58 index: 73 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:05.524876 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 59 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:05.524981 20891 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 59 FOLLOWER]: Leader election lost for term 59. Reason: could not achieve majority
I20251212 03:12:05.532670 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 59 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 58. Current term is 59. Ops: []
I20251212 03:12:05.532974 20880 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 58.73, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20251212 03:12:05.533116 20890 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 58 LEADER]: Stepping down as leader of term 58
I20251212 03:12:05.533182 20890 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 58 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:05.533252 20890 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 58.73, Last appended by leader: 73, Current term: 58, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:05.533341 20890 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 58 FOLLOWER]: Advancing to term 59
I20251212 03:12:05.537612 20887 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 58 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:05.537674 20887 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 58 FOLLOWER]: Advancing to term 59
I20251212 03:12:05.538200 20887 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 59 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:05.538368 20887 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 59 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:05.538559 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 59 candidate_status { last_received { term: 58 index: 73 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:05.538569 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 59 candidate_status { last_received { term: 58 index: 73 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:05.538627 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 59 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 59: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:05.539161 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 59 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 59.
I20251212 03:12:05.539319 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 59 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:05.539418 20887 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 59 FOLLOWER]: Leader election won for term 59
I20251212 03:12:05.539495 20887 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 59 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:05.539559 20887 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 58.73, Last appended by leader: 73, Current term: 59, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:05.539999 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 58 to 59, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 59 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:05.631634 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 59 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 58 index: 73. Preceding OpId from leader: term: 59 index: 74. (index mismatch)
I20251212 03:12:05.631970 20887 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20251212 03:12:05.633229 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 59 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 58 index: 73. Preceding OpId from leader: term: 59 index: 74. (index mismatch)
I20251212 03:12:05.633459 20895 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20251212 03:12:05.914233 20891 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 59 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:05.914337 20891 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 59 FOLLOWER]: Advancing to term 60
I20251212 03:12:05.914604 20890 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 59 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:05.914682 20890 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 59 FOLLOWER]: Advancing to term 60
I20251212 03:12:05.915091 20891 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 60 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:05.915251 20891 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 60 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:05.915323 20890 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 60 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:05.915470 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 60 candidate_status { last_received { term: 59 index: 74 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:05.915477 20890 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 60 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:05.915573 20682 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 60 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 in current term 60: Already voted for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in this term.
I20251212 03:12:05.915477 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 60 candidate_status { last_received { term: 59 index: 74 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:05.915627 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 60 candidate_status { last_received { term: 59 index: 74 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:05.915757 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 60 candidate_status { last_received { term: 59 index: 74 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:05.915803 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 60 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:05.915839 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 60 FOLLOWER]: Leader election vote request: Denying vote to candidate d3fe965dfda94fd4bf3f61a5c0452d63 in current term 60: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:05.915911 20891 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 60 FOLLOWER]: Leader election lost for term 60. Reason: could not achieve majority
I20251212 03:12:05.915970 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 60 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:05.916034 20890 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 60 FOLLOWER]: Leader election lost for term 60. Reason: could not achieve majority
I20251212 03:12:05.934008 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 60 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 59. Current term is 60. Ops: []
I20251212 03:12:05.934279 20895 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 59.74, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20251212 03:12:05.934412 20895 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 59 LEADER]: Stepping down as leader of term 59
I20251212 03:12:05.934449 20895 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 59 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:05.934525 20895 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 59.74, Last appended by leader: 74, Current term: 59, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:05.934769 20895 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 59 FOLLOWER]: Advancing to term 60
I20251212 03:12:06.039618 20891 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 60 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:06.039723 20891 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 60 FOLLOWER]: Advancing to term 61
I20251212 03:12:06.040408 20891 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 61 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:06.040663 20891 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 61 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:06.040906 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 61 candidate_status { last_received { term: 59 index: 74 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:06.040906 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 61 candidate_status { last_received { term: 59 index: 74 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:06.041028 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 60 FOLLOWER]: Advancing to term 61
I20251212 03:12:06.041057 20758 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 60 FOLLOWER]: Advancing to term 61
I20251212 03:12:06.041783 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 61 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 61.
I20251212 03:12:06.041783 20758 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 61 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 61.
I20251212 03:12:06.042028 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 61 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters:
I20251212 03:12:06.042133 20891 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 61 FOLLOWER]: Leader election won for term 61
I20251212 03:12:06.042367 20891 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 61 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:06.042454 20891 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 59.74, Last appended by leader: 74, Current term: 61, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:06.042986 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 59 to 61, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 61 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:06.123379 20758 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 61 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 59 index: 74. Preceding OpId from leader: term: 61 index: 75. (index mismatch)
I20251212 03:12:06.123705 20891 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20251212 03:12:06.124827 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 61 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 59 index: 74. Preceding OpId from leader: term: 61 index: 75. (index mismatch)
I20251212 03:12:06.125130 20900 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20251212 03:12:06.319515 20895 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 61 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:06.319628 20895 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 61 FOLLOWER]: Advancing to term 62
I20251212 03:12:06.320326 20895 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 62 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:06.320492 20895 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 62 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:06.320736 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 62 candidate_status { last_received { term: 61 index: 75 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:06.320747 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 62 candidate_status { last_received { term: 61 index: 75 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:06.321008 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 62 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:06.321158 20895 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 62 FOLLOWER]: Leader election lost for term 62. Reason: could not achieve majority
I20251212 03:12:06.324982 20901 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 61 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:06.325042 20901 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 61 FOLLOWER]: Advancing to term 62
I20251212 03:12:06.325685 20901 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 62 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:06.325829 20901 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 62 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:06.325990 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 62 candidate_status { last_received { term: 61 index: 75 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:06.326032 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 62 candidate_status { last_received { term: 61 index: 75 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:06.326054 20758 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 62 FOLLOWER]: Leader election vote request: Denying vote to candidate d3fe965dfda94fd4bf3f61a5c0452d63 in current term 62: Already voted for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in this term.
I20251212 03:12:06.326299 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 62 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:06.326386 20901 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 62 FOLLOWER]: Leader election lost for term 62. Reason: could not achieve majority
I20251212 03:12:06.330132 20758 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 62 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 61. Current term is 62. Ops: []
I20251212 03:12:06.330375 20896 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 61.75, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20251212 03:12:06.330492 20896 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 61 LEADER]: Stepping down as leader of term 61
I20251212 03:12:06.330530 20896 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 61 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:06.330590 20896 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 61.75, Last appended by leader: 75, Current term: 61, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:06.330657 20896 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 61 FOLLOWER]: Advancing to term 62
I20251212 03:12:06.446696 20895 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 62 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:06.446805 20895 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 62 FOLLOWER]: Advancing to term 63
I20251212 03:12:06.447525 20895 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 63 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:06.447710 20895 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 63 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:06.447959 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 63 candidate_status { last_received { term: 61 index: 75 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:06.447959 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 63 candidate_status { last_received { term: 61 index: 75 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:06.448087 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 62 FOLLOWER]: Advancing to term 63
I20251212 03:12:06.448103 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 62 FOLLOWER]: Advancing to term 63
I20251212 03:12:06.448796 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 63 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 63.
I20251212 03:12:06.448796 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 63 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 63.
I20251212 03:12:06.449051 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 63 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters:
I20251212 03:12:06.449189 20895 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 63 FOLLOWER]: Leader election won for term 63
I20251212 03:12:06.449301 20895 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 63 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:06.449347 20895 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 61.75, Last appended by leader: 75, Current term: 63, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:06.449867 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 61 to 63, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 63 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:06.496110 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:06.532363 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 63 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 61 index: 75. Preceding OpId from leader: term: 63 index: 76. (index mismatch)
I20251212 03:12:06.532711 20895 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20251212 03:12:06.533828 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 63 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 61 index: 75. Preceding OpId from leader: term: 63 index: 76. (index mismatch)
I20251212 03:12:06.534049 20902 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20251212 03:12:06.727996 20890 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 63 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:06.728101 20890 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 63 FOLLOWER]: Advancing to term 64
I20251212 03:12:06.728857 20890 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 64 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:06.729035 20890 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 64 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:06.729257 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 64 candidate_status { last_received { term: 63 index: 76 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:06.729257 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 64 candidate_status { last_received { term: 63 index: 76 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:06.729549 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 64 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:06.729645 20890 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 64 FOLLOWER]: Leader election lost for term 64. Reason: could not achieve majority
I20251212 03:12:06.740794 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 64 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 63. Current term is 64. Ops: []
I20251212 03:12:06.741168 20903 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 63.76, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20251212 03:12:06.741328 20903 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 63 LEADER]: Stepping down as leader of term 63
I20251212 03:12:06.741405 20903 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 63 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:06.741479 20903 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 63.76, Last appended by leader: 76, Current term: 63, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:06.741565 20903 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 63 FOLLOWER]: Advancing to term 64
I20251212 03:12:06.798381 20896 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 63 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:06.798491 20896 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 63 FOLLOWER]: Advancing to term 64
I20251212 03:12:06.799952 20896 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 64 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:06.800114 20896 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 64 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:06.800349 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 64 candidate_status { last_received { term: 63 index: 76 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:06.800369 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 64 candidate_status { last_received { term: 63 index: 76 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:06.800500 20682 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 64 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 in current term 64: Already voted for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in this term.
I20251212 03:12:06.801036 20758 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 64 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 64.
I20251212 03:12:06.801247 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 64 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:06.801353 20896 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 64 FOLLOWER]: Leader election won for term 64
I20251212 03:12:06.801409 20896 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 64 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:06.801461 20896 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 63.76, Last appended by leader: 76, Current term: 64, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:06.801997 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 63 to 64, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 64 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:06.871181 20890 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 64 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:06.871284 20890 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 64 FOLLOWER]: Advancing to term 65
I20251212 03:12:06.871944 20890 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 65 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:06.872098 20890 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 65 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:06.872326 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 65 candidate_status { last_received { term: 63 index: 76 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:06.872327 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 65 candidate_status { last_received { term: 63 index: 76 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:06.872473 20758 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 64 FOLLOWER]: Advancing to term 65
I20251212 03:12:06.873075 20758 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 65 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 65.
I20251212 03:12:06.873238 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 65 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:06.873328 20890 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 65 FOLLOWER]: Leader election won for term 65
I20251212 03:12:06.873379 20890 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 65 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:06.873412 20890 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 63.76, Last appended by leader: 76, Current term: 65, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:06.873848 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 64 to 65, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 65 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:06.881289 20758 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 65 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 64. Current term is 65. Ops: []
I20251212 03:12:06.881546 20896 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20251212 03:12:06.881642 20896 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 64 LEADER]: Stepping down as leader of term 64
I20251212 03:12:06.881686 20896 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 64 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:06.881754 20896 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 64.77, Last appended by leader: 77, Current term: 64, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:06.881829 20896 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 64 FOLLOWER]: Advancing to term 65
I20251212 03:12:06.968286 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 65 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 64 index: 77. Preceding OpId from leader: term: 65 index: 77. (term mismatch)
I20251212 03:12:06.968359 20605 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting all ops after (but not including) 76
I20251212 03:12:06.968375 20605 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting uncommitted NO_OP operation due to leader change: 64.77
I20251212 03:12:06.968390 20605 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 65 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:06.968638 20890 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20251212 03:12:06.969667 20758 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 65 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 63 index: 76. Preceding OpId from leader: term: 65 index: 77. (index mismatch)
I20251212 03:12:06.969856 20901 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20251212 03:12:07.158783 20896 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 65 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:07.158871 20896 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 65 FOLLOWER]: Advancing to term 66
I20251212 03:12:07.159582 20896 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 66 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:07.159761 20896 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 66 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:07.159991 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 66 candidate_status { last_received { term: 65 index: 77 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:07.159987 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 66 candidate_status { last_received { term: 65 index: 77 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:07.160254 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 66 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:07.160344 20896 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 66 FOLLOWER]: Leader election lost for term 66. Reason: could not achieve majority
I20251212 03:12:07.161756 20895 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 65 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:07.161821 20895 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 65 FOLLOWER]: Advancing to term 66
I20251212 03:12:07.162431 20895 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 66 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:07.162554 20758 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 66 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 65. Current term is 66. Ops: []
I20251212 03:12:07.162571 20895 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 66 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:07.162703 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 66 candidate_status { last_received { term: 65 index: 77 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:07.162774 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 66 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 66: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:07.162783 20904 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 65.77, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20251212 03:12:07.162930 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 66 candidate_status { last_received { term: 65 index: 77 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:07.162979 20904 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 65 LEADER]: Stepping down as leader of term 65
I20251212 03:12:07.163060 20904 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 65 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:07.163117 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 66 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:07.163129 20904 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 65.77, Last appended by leader: 77, Current term: 65, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:07.163223 20895 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 66 FOLLOWER]: Leader election lost for term 66. Reason: could not achieve majority
I20251212 03:12:07.163231 20904 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 65 FOLLOWER]: Advancing to term 66
I20251212 03:12:07.261289 20896 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 66 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:07.261394 20896 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 66 FOLLOWER]: Advancing to term 67
I20251212 03:12:07.262068 20896 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 67 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:07.262290 20896 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 67 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:07.262450 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 67 candidate_status { last_received { term: 65 index: 77 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:07.262501 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 67 candidate_status { last_received { term: 65 index: 77 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:07.262521 20758 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 66 FOLLOWER]: Advancing to term 67
I20251212 03:12:07.262614 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 66 FOLLOWER]: Advancing to term 67
I20251212 03:12:07.263139 20758 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 67 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 67.
I20251212 03:12:07.263201 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 67 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 67.
I20251212 03:12:07.263351 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 67 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters:
I20251212 03:12:07.263453 20896 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 67 FOLLOWER]: Leader election won for term 67
I20251212 03:12:07.263514 20896 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 67 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:07.263566 20896 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 65.77, Last appended by leader: 77, Current term: 67, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:07.264107 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 65 to 67, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 67 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:07.360654 20758 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 67 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 65 index: 77. Preceding OpId from leader: term: 67 index: 78. (index mismatch)
I20251212 03:12:07.361003 20896 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20251212 03:12:07.362694 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 67 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 65 index: 77. Preceding OpId from leader: term: 67 index: 78. (index mismatch)
I20251212 03:12:07.362933 20891 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s
I20251212 03:12:07.463054 20895 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 67 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:07.463160 20895 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 67 FOLLOWER]: Advancing to term 68
I20251212 03:12:07.463876 20895 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 68 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:07.464037 20895 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 68 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:07.464272 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 68 candidate_status { last_received { term: 67 index: 78 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:07.464291 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 68 candidate_status { last_received { term: 67 index: 78 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:07.464572 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 68 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:07.464728 20895 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 68 FOLLOWER]: Leader election lost for term 68. Reason: could not achieve majority
I20251212 03:12:07.485078 20758 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 68 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 67. Current term is 68. Ops: []
I20251212 03:12:07.485351 20896 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 67.78, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20251212 03:12:07.485481 20891 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 67 LEADER]: Stepping down as leader of term 67
I20251212 03:12:07.485533 20891 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 67 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:07.485596 20891 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 67.78, Last appended by leader: 78, Current term: 67, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:07.485694 20891 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 67 FOLLOWER]: Advancing to term 68
I20251212 03:12:07.496377 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:07.547811 20901 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 67 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:07.547909 20901 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 67 FOLLOWER]: Advancing to term 68
I20251212 03:12:07.548566 20901 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 68 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:07.548730 20901 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 68 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:07.548991 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 68 candidate_status { last_received { term: 67 index: 78 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:07.548991 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 68 candidate_status { last_received { term: 67 index: 78 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:07.549130 20758 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 68 FOLLOWER]: Leader election vote request: Denying vote to candidate d3fe965dfda94fd4bf3f61a5c0452d63 in current term 68: Already voted for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in this term.
I20251212 03:12:07.549695 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 68 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 68.
I20251212 03:12:07.549904 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 68 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:07.550007 20901 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 68 FOLLOWER]: Leader election won for term 68
I20251212 03:12:07.550060 20901 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 68 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:07.550101 20901 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 67.78, Last appended by leader: 78, Current term: 68, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:07.550647 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 67 to 68, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 68 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:07.570500 20895 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 68 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:07.570595 20895 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 68 FOLLOWER]: Advancing to term 69
I20251212 03:12:07.571252 20895 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 69 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:07.571445 20895 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 69 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:07.571673 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 69 candidate_status { last_received { term: 67 index: 78 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:07.571673 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 69 candidate_status { last_received { term: 67 index: 78 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:07.571834 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 68 FOLLOWER]: Advancing to term 69
I20251212 03:12:07.572376 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 69 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 69.
I20251212 03:12:07.572598 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 69 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:07.572723 20895 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 69 FOLLOWER]: Leader election won for term 69
I20251212 03:12:07.572860 20895 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 69 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:07.572906 20895 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 67.78, Last appended by leader: 78, Current term: 69, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:07.573366 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 68 to 69, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 69 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:07.650184 20758 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 69 LEADER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 68. Current term is 69. Ops: []
I20251212 03:12:07.650550 20901 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20251212 03:12:07.650671 20901 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer f894b0d567ab4a2ebdcb416d04ef48e5 log is divergent from this leader: its last log entry 69.79 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 78
I20251212 03:12:07.650725 20904 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 68 LEADER]: Stepping down as leader of term 68
I20251212 03:12:07.650764 20904 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 68 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:07.650816 20904 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 68.79, Last appended by leader: 79, Current term: 68, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:07.650879 20904 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 68 FOLLOWER]: Advancing to term 69
I20251212 03:12:07.662845 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 69 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 67 index: 78. Preceding OpId from leader: term: 69 index: 79. (index mismatch)
I20251212 03:12:07.663213 20895 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20251212 03:12:07.664469 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 69 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 68 index: 79. Preceding OpId from leader: term: 69 index: 79. (term mismatch)
I20251212 03:12:07.664534 20682 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting all ops after (but not including) 78
I20251212 03:12:07.664559 20682 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting uncommitted NO_OP operation due to leader change: 68.79
I20251212 03:12:07.664577 20682 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 69 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:07.664752 20905 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20251212 03:12:07.765256 20891 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 69 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:07.765410 20891 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 69 FOLLOWER]: Advancing to term 70
I20251212 03:12:07.766067 20891 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 70 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:07.766266 20891 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 70 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:07.766489 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 70 candidate_status { last_received { term: 69 index: 79 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:07.766494 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 70 candidate_status { last_received { term: 69 index: 79 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:07.766625 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 69 FOLLOWER]: Advancing to term 70
I20251212 03:12:07.767295 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 70 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 70.
I20251212 03:12:07.767527 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 70 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:07.767645 20891 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 70 FOLLOWER]: Leader election won for term 70
I20251212 03:12:07.767721 20891 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 70 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:07.767782 20891 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 69.79, Last appended by leader: 79, Current term: 70, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:07.768352 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 69 to 70, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 70 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:07.775583 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 70 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 69. Current term is 70. Ops: []
I20251212 03:12:07.775810 20906 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 69.79, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20251212 03:12:07.776036 20895 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 69 LEADER]: Stepping down as leader of term 69
I20251212 03:12:07.776083 20895 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 69 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:07.776139 20895 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 69.79, Last appended by leader: 79, Current term: 69, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:07.776222 20895 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 69 FOLLOWER]: Advancing to term 70
I20251212 03:12:07.860031 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 70 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 69 index: 79. Preceding OpId from leader: term: 70 index: 80. (index mismatch)
I20251212 03:12:07.860440 20891 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20251212 03:12:07.861887 20758 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 70 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 69 index: 79. Preceding OpId from leader: term: 70 index: 80. (index mismatch)
I20251212 03:12:07.862109 20891 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20251212 03:12:08.058928 20904 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 70 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:08.059039 20904 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 70 FOLLOWER]: Advancing to term 71
I20251212 03:12:08.059808 20904 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 71 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:08.060073 20904 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 71 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:08.060240 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 71 candidate_status { last_received { term: 70 index: 80 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:08.060199 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 71 candidate_status { last_received { term: 70 index: 80 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:08.060509 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 71 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:08.060635 20904 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 71 FOLLOWER]: Leader election lost for term 71. Reason: could not achieve majority
I20251212 03:12:08.071137 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 71 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 70. Current term is 71. Ops: []
I20251212 03:12:08.071489 20891 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 70.80, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20251212 03:12:08.071645 20891 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 70 LEADER]: Stepping down as leader of term 70
I20251212 03:12:08.071681 20891 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 70 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:08.071758 20891 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 70.80, Last appended by leader: 80, Current term: 70, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:08.071851 20891 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 70 FOLLOWER]: Advancing to term 71
I20251212 03:12:08.136768 20906 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 70 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:08.136878 20906 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 70 FOLLOWER]: Advancing to term 71
I20251212 03:12:08.137620 20906 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 71 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:08.137789 20906 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 71 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:08.138026 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 71 candidate_status { last_received { term: 70 index: 80 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:08.138031 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 71 candidate_status { last_received { term: 70 index: 80 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:08.138154 20682 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 71 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 71: Already voted for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in this term.
I20251212 03:12:08.138686 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 71 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 71.
I20251212 03:12:08.138859 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 71 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:08.138939 20906 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 71 FOLLOWER]: Leader election won for term 71
I20251212 03:12:08.138995 20906 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 71 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:08.139053 20906 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 70.80, Last appended by leader: 80, Current term: 71, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:08.139523 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 70 to 71, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 71 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:08.198189 20904 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 71 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:08.198383 20904 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 71 FOLLOWER]: Advancing to term 72
I20251212 03:12:08.199069 20904 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 72 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:08.199250 20904 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 72 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:08.199492 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 72 candidate_status { last_received { term: 70 index: 80 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:08.199491 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 72 candidate_status { last_received { term: 70 index: 80 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:08.199641 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 71 FOLLOWER]: Advancing to term 72
I20251212 03:12:08.200202 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 72 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 72.
I20251212 03:12:08.200477 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 72 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:08.200629 20904 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 72 FOLLOWER]: Leader election won for term 72
I20251212 03:12:08.200755 20904 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 72 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:08.200830 20904 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 70.80, Last appended by leader: 80, Current term: 72, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:08.201269 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 71 to 72, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 72 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:08.224465 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 72 LEADER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 71. Current term is 72. Ops: []
I20251212 03:12:08.224751 20906 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20251212 03:12:08.224860 20906 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer d3fe965dfda94fd4bf3f61a5c0452d63 log is divergent from this leader: its last log entry 72.81 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 80
I20251212 03:12:08.224893 20895 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 71 LEADER]: Stepping down as leader of term 71
I20251212 03:12:08.224934 20895 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 71 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:08.224988 20895 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 71.81, Last appended by leader: 81, Current term: 71, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:08.225070 20895 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 71 FOLLOWER]: Advancing to term 72
I20251212 03:12:08.313840 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 72 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 70 index: 80. Preceding OpId from leader: term: 72 index: 81. (index mismatch)
I20251212 03:12:08.314136 20904 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20251212 03:12:08.315166 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 72 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 71 index: 81. Preceding OpId from leader: term: 72 index: 81. (term mismatch)
I20251212 03:12:08.315223 20759 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting all ops after (but not including) 80
I20251212 03:12:08.315248 20759 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting uncommitted NO_OP operation due to leader change: 71.81
I20251212 03:12:08.315266 20759 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 72 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:08.315438 20907 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20251212 03:12:08.415851 20891 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 72 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:08.415951 20891 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 72 FOLLOWER]: Advancing to term 73
I20251212 03:12:08.416644 20891 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 73 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:08.416818 20891 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 73 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:08.417016 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 73 candidate_status { last_received { term: 72 index: 81 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:08.417021 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 73 candidate_status { last_received { term: 72 index: 81 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:08.417258 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 73 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:08.417376 20891 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 73 FOLLOWER]: Leader election lost for term 73. Reason: could not achieve majority
I20251212 03:12:08.423297 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 73 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 72. Current term is 73. Ops: []
I20251212 03:12:08.423557 20908 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 72.81, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20251212 03:12:08.423673 20908 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 72 LEADER]: Stepping down as leader of term 72
I20251212 03:12:08.423693 20908 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 72 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:08.423767 20908 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 72.81, Last appended by leader: 81, Current term: 72, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:08.423856 20908 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 72 FOLLOWER]: Advancing to term 73
I20251212 03:12:08.496639 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:08.511842 20906 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 72 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:08.511957 20906 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 72 FOLLOWER]: Advancing to term 73
I20251212 03:12:08.512833 20906 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 73 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:08.513178 20906 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 73 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:08.513527 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 73 candidate_status { last_received { term: 72 index: 81 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:08.513527 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 73 candidate_status { last_received { term: 72 index: 81 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:08.513669 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 73 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 73: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:08.514277 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 73 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 73.
I20251212 03:12:08.514519 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 73 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:08.514623 20906 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 73 FOLLOWER]: Leader election won for term 73
I20251212 03:12:08.514673 20906 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 73 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:08.514739 20906 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 72.81, Last appended by leader: 81, Current term: 73, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:08.515209 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 72 to 73, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 73 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:08.529888 20891 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 73 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:08.529985 20891 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 73 FOLLOWER]: Advancing to term 74
I20251212 03:12:08.530777 20891 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 74 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:08.530951 20891 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 74 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:08.531206 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 74 candidate_status { last_received { term: 72 index: 81 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:08.531206 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 74 candidate_status { last_received { term: 72 index: 81 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:08.531329 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 73 FOLLOWER]: Advancing to term 74
I20251212 03:12:08.531949 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 74 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 74.
I20251212 03:12:08.532160 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 74 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:08.532294 20891 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 74 FOLLOWER]: Leader election won for term 74
I20251212 03:12:08.532418 20891 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 74 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:08.532478 20891 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 72.81, Last appended by leader: 81, Current term: 74, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:08.532927 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 73 to 74, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 74 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:08.596549 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 74 LEADER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 73. Current term is 74. Ops: []
I20251212 03:12:08.596895 20906 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20251212 03:12:08.597016 20906 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer e9d27948dced4eba8f8228ddc4529393 log is divergent from this leader: its last log entry 74.82 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 81
I20251212 03:12:08.597060 20895 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 73 LEADER]: Stepping down as leader of term 73
I20251212 03:12:08.597101 20895 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 73 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:08.597157 20895 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 73.82, Last appended by leader: 82, Current term: 73, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:08.597250 20895 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 73 FOLLOWER]: Advancing to term 74
I20251212 03:12:08.627679 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 74 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 72 index: 81. Preceding OpId from leader: term: 74 index: 82. (index mismatch)
I20251212 03:12:08.627964 20891 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20251212 03:12:08.628988 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 74 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 73 index: 82. Preceding OpId from leader: term: 74 index: 82. (term mismatch)
I20251212 03:12:08.629045 20759 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting all ops after (but not including) 81
I20251212 03:12:08.629063 20759 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting uncommitted NO_OP operation due to leader change: 73.82
I20251212 03:12:08.629086 20759 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 74 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:08.629231 20891 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20251212 03:12:08.805833 20908 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 74 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:08.805920 20908 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 74 FOLLOWER]: Advancing to term 75
I20251212 03:12:08.806640 20908 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 75 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:08.806797 20908 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 75 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:08.806977 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 75 candidate_status { last_received { term: 74 index: 82 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:08.807082 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 75 candidate_status { last_received { term: 74 index: 82 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:08.807332 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 75 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:08.807459 20908 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 75 FOLLOWER]: Leader election lost for term 75. Reason: could not achieve majority
I20251212 03:12:08.816886 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 75 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 74. Current term is 75. Ops: []
I20251212 03:12:08.817139 20896 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 74.82, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20251212 03:12:08.817289 20891 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 74 LEADER]: Stepping down as leader of term 74
I20251212 03:12:08.817343 20891 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 74 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:08.817390 20891 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 74.82, Last appended by leader: 82, Current term: 74, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:08.817513 20891 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 74 FOLLOWER]: Advancing to term 75
I20251212 03:12:08.905123 20895 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 74 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:08.905231 20895 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 74 FOLLOWER]: Advancing to term 75
I20251212 03:12:08.905941 20895 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 75 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:08.906109 20895 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 75 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:08.906374 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 75 candidate_status { last_received { term: 74 index: 82 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:08.906422 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 75 candidate_status { last_received { term: 74 index: 82 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:08.906541 20682 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 75 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 75: Already voted for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in this term.
I20251212 03:12:08.906997 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 75 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 75.
I20251212 03:12:08.907265 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 75 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:08.907486 20895 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 75 FOLLOWER]: Leader election won for term 75
I20251212 03:12:08.907560 20895 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 75 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:08.907618 20895 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 74.82, Last appended by leader: 82, Current term: 75, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:08.908110 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 74 to 75, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 75 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:08.938899 20908 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 75 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:08.938998 20908 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 75 FOLLOWER]: Advancing to term 76
I20251212 03:12:08.939673 20908 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 76 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:08.939841 20908 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 76 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:08.940080 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 76 candidate_status { last_received { term: 74 index: 82 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:08.940083 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 76 candidate_status { last_received { term: 74 index: 82 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:08.940151 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 75 FOLLOWER]: Advancing to term 76
I20251212 03:12:08.940749 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 76 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 76.
I20251212 03:12:08.940965 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 76 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:08.941046 20908 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 76 FOLLOWER]: Leader election won for term 76
I20251212 03:12:08.941143 20908 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 76 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:08.941182 20908 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 74.82, Last appended by leader: 82, Current term: 76, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:08.941637 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 75 to 76, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 76 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:09.017004 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 76 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 74 index: 82. Preceding OpId from leader: term: 76 index: 83. (index mismatch)
I20251212 03:12:09.017338 20908 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20251212 03:12:09.018692 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 76 LEADER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 75. Current term is 76. Ops: []
I20251212 03:12:09.018692 20759 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 75 LEADER]: Stepping down as leader of term 75
I20251212 03:12:09.018788 20759 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 75 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:09.018838 20759 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 75.83, Last appended by leader: 83, Current term: 75, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:09.018906 20895 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 83, Last known committed idx: 83, Time since last communication: 0.000s
I20251212 03:12:09.019001 20895 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Peer d3fe965dfda94fd4bf3f61a5c0452d63 log is divergent from this leader: its last log entry 76.83 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 83
I20251212 03:12:09.019070 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 75 FOLLOWER]: Advancing to term 76
I20251212 03:12:09.019738 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 76 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 75 index: 83. Preceding OpId from leader: term: 76 index: 83. (term mismatch)
I20251212 03:12:09.019778 20759 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting all ops after (but not including) 82
I20251212 03:12:09.019797 20759 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting uncommitted NO_OP operation due to leader change: 75.83
I20251212 03:12:09.019817 20759 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 76 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:09.020138 20910 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20251212 03:12:09.289420 20906 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 76 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:09.289532 20906 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 76 FOLLOWER]: Advancing to term 77
I20251212 03:12:09.290307 20906 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 77 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:09.290488 20906 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 77 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:09.290716 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 77 candidate_status { last_received { term: 76 index: 83 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:09.290716 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 77 candidate_status { last_received { term: 76 index: 83 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:09.290992 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 77 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:09.291085 20906 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 77 FOLLOWER]: Leader election lost for term 77. Reason: could not achieve majority
I20251212 03:12:09.297222 20891 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 76 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:09.297304 20891 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 76 FOLLOWER]: Advancing to term 77
I20251212 03:12:09.297940 20891 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 77 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:09.298085 20891 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 77 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:09.298313 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 77 candidate_status { last_received { term: 76 index: 83 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:09.298314 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 77 candidate_status { last_received { term: 76 index: 83 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:09.298429 20759 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 77 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 in current term 77: Already voted for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in this term.
I20251212 03:12:09.298607 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 77 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:09.298696 20891 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 77 FOLLOWER]: Leader election lost for term 77. Reason: could not achieve majority
I20251212 03:12:09.306147 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 77 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 76. Current term is 77. Ops: []
I20251212 03:12:09.306457 20908 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 76.83, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20251212 03:12:09.306572 20908 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 76 LEADER]: Stepping down as leader of term 76
I20251212 03:12:09.306607 20908 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 76 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:09.306667 20908 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 76.83, Last appended by leader: 83, Current term: 76, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:09.306746 20908 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 76 FOLLOWER]: Advancing to term 77
I20251212 03:12:09.411101 20906 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 77 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:09.411211 20906 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 77 FOLLOWER]: Advancing to term 78
I20251212 03:12:09.412020 20906 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 78 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:09.412240 20906 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 78 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:09.412580 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 78 candidate_status { last_received { term: 76 index: 83 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:09.412580 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 78 candidate_status { last_received { term: 76 index: 83 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:09.412735 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 77 FOLLOWER]: Advancing to term 78
I20251212 03:12:09.412742 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 77 FOLLOWER]: Advancing to term 78
I20251212 03:12:09.413453 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 78 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 78.
I20251212 03:12:09.413453 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 78 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 78.
I20251212 03:12:09.413717 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 78 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters:
I20251212 03:12:09.413827 20906 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 78 FOLLOWER]: Leader election won for term 78
I20251212 03:12:09.413884 20906 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 78 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:09.413929 20906 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 76.83, Last appended by leader: 83, Current term: 78, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:09.414378 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 76 to 78, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 78 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:09.496898 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:09.514074 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 78 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 76 index: 83. Preceding OpId from leader: term: 78 index: 84. (index mismatch)
I20251212 03:12:09.514529 20906 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20251212 03:12:09.515633 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 78 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 76 index: 83. Preceding OpId from leader: term: 78 index: 84. (index mismatch)
I20251212 03:12:09.515851 20906 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20251212 03:12:09.616566 20891 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 78 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:09.616674 20891 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 78 FOLLOWER]: Advancing to term 79
I20251212 03:12:09.617338 20891 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 79 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:09.617506 20891 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 79 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:09.617781 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 79 candidate_status { last_received { term: 78 index: 84 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:09.617781 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 79 candidate_status { last_received { term: 78 index: 84 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:09.618115 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 79 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:09.618242 20891 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 79 FOLLOWER]: Leader election lost for term 79. Reason: could not achieve majority
I20251212 03:12:09.637840 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 79 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 78. Current term is 79. Ops: []
I20251212 03:12:09.638118 20911 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 78.84, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20251212 03:12:09.638268 20911 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 78 LEADER]: Stepping down as leader of term 78
I20251212 03:12:09.638305 20911 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 78 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:09.638363 20911 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 78.84, Last appended by leader: 84, Current term: 78, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:09.638461 20911 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 78 FOLLOWER]: Advancing to term 79
I20251212 03:12:09.701741 20910 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 78 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:09.701846 20910 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 78 FOLLOWER]: Advancing to term 79
I20251212 03:12:09.702760 20910 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 79 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:09.702961 20910 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 79 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:09.703183 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 79 candidate_status { last_received { term: 78 index: 84 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:09.703208 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 79 candidate_status { last_received { term: 78 index: 84 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:09.703306 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 79 FOLLOWER]: Leader election vote request: Denying vote to candidate d3fe965dfda94fd4bf3f61a5c0452d63 in current term 79: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:09.703900 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 79 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 79.
I20251212 03:12:09.704097 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 79 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:09.704193 20910 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 79 FOLLOWER]: Leader election won for term 79
I20251212 03:12:09.704264 20910 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 79 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:09.704315 20910 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 78.84, Last appended by leader: 84, Current term: 79, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:09.704949 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 78 to 79, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 79 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:09.749784 20891 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 79 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:09.749895 20891 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 79 FOLLOWER]: Advancing to term 80
I20251212 03:12:09.750792 20891 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 80 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:09.750963 20891 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 80 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:09.751176 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 80 candidate_status { last_received { term: 78 index: 84 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:09.751221 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 80 candidate_status { last_received { term: 78 index: 84 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:09.751237 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 79 FOLLOWER]: Advancing to term 80
I20251212 03:12:09.752123 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 80 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 80.
I20251212 03:12:09.752279 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 80 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:09.752377 20891 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 80 FOLLOWER]: Leader election won for term 80
I20251212 03:12:09.752503 20891 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 80 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:09.752578 20891 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 78.84, Last appended by leader: 84, Current term: 80, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:09.753139 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 79 to 80, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 80 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:09.804225 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 80 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 79. Current term is 80. Ops: []
I20251212 03:12:09.804598 20910 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20251212 03:12:09.804720 20910 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 79 LEADER]: Stepping down as leader of term 79
I20251212 03:12:09.804739 20910 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 79 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:09.804801 20910 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 79.85, Last appended by leader: 85, Current term: 79, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:09.804885 20910 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 79 FOLLOWER]: Advancing to term 80
I20251212 03:12:09.834676 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 80 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 78 index: 84. Preceding OpId from leader: term: 80 index: 85. (index mismatch)
I20251212 03:12:09.834956 20891 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20251212 03:12:09.836097 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 80 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 79 index: 85. Preceding OpId from leader: term: 80 index: 85. (term mismatch)
I20251212 03:12:09.836158 20682 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting all ops after (but not including) 84
I20251212 03:12:09.836179 20682 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting uncommitted NO_OP operation due to leader change: 79.85
I20251212 03:12:09.836200 20682 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 80 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:09.836447 20891 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20251212 03:12:10.040452 20910 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 80 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:10.040563 20910 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 80 FOLLOWER]: Advancing to term 81
I20251212 03:12:10.041262 20910 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 81 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:10.041472 20910 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 81 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:10.041646 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 81 candidate_status { last_received { term: 80 index: 85 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:10.041738 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 81 candidate_status { last_received { term: 80 index: 85 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:10.041963 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 81 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:10.042052 20910 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 81 FOLLOWER]: Leader election lost for term 81. Reason: could not achieve majority
I20251212 03:12:10.046944 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 81 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 80. Current term is 81. Ops: []
I20251212 03:12:10.047209 20891 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 80.85, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20251212 03:12:10.047328 20891 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 80 LEADER]: Stepping down as leader of term 80
I20251212 03:12:10.047354 20891 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 80 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:10.047422 20891 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 80.85, Last appended by leader: 85, Current term: 80, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:10.047497 20891 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 80 FOLLOWER]: Advancing to term 81
I20251212 03:12:10.126148 20911 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 80 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:10.126271 20911 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 80 FOLLOWER]: Advancing to term 81
I20251212 03:12:10.126996 20911 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 81 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:10.127168 20911 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 81 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:10.127398 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 81 candidate_status { last_received { term: 80 index: 85 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:10.127436 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 81 candidate_status { last_received { term: 80 index: 85 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:10.127506 20682 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 81 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 81: Already voted for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in this term.
I20251212 03:12:10.128058 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 81 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 81.
I20251212 03:12:10.128247 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 81 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:10.128334 20911 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 81 FOLLOWER]: Leader election won for term 81
I20251212 03:12:10.128392 20911 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 81 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:10.128460 20911 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 80.85, Last appended by leader: 85, Current term: 81, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:10.128880 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 80 to 81, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 81 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:10.163530 20910 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 81 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:10.163652 20910 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 81 FOLLOWER]: Advancing to term 82
I20251212 03:12:10.164305 20910 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 82 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:10.164470 20910 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 82 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:10.164721 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 82 candidate_status { last_received { term: 80 index: 85 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:10.164721 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 82 candidate_status { last_received { term: 80 index: 85 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:10.164821 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 81 FOLLOWER]: Advancing to term 82
I20251212 03:12:10.165387 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 82 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 82.
I20251212 03:12:10.165551 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 82 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:10.165685 20910 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 82 FOLLOWER]: Leader election won for term 82
I20251212 03:12:10.165764 20910 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 82 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:10.165813 20910 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 80.85, Last appended by leader: 85, Current term: 82, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:10.166311 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 81 to 82, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 82 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:10.204563 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 82 LEADER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 81. Current term is 82. Ops: []
I20251212 03:12:10.204844 20911 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20251212 03:12:10.204962 20911 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer d3fe965dfda94fd4bf3f61a5c0452d63 log is divergent from this leader: its last log entry 82.86 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 85
I20251212 03:12:10.205004 20911 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 81 LEADER]: Stepping down as leader of term 81
I20251212 03:12:10.205031 20911 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 81 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:10.205093 20911 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 81.86, Last appended by leader: 86, Current term: 81, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:10.205173 20911 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 81 FOLLOWER]: Advancing to term 82
I20251212 03:12:10.262010 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 82 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 81 index: 86. Preceding OpId from leader: term: 82 index: 86. (term mismatch)
I20251212 03:12:10.262107 20759 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting all ops after (but not including) 85
I20251212 03:12:10.262137 20759 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting uncommitted NO_OP operation due to leader change: 81.86
I20251212 03:12:10.262159 20759 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 82 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:10.262458 20910 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20251212 03:12:10.263783 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 82 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 80 index: 85. Preceding OpId from leader: term: 82 index: 86. (index mismatch)
I20251212 03:12:10.264015 20914 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20251212 03:12:10.364920 20913 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 82 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:10.365015 20913 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 82 FOLLOWER]: Advancing to term 83
I20251212 03:12:10.365773 20913 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 83 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:10.365976 20913 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 83 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:10.366189 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 83 candidate_status { last_received { term: 82 index: 86 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:10.366189 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 83 candidate_status { last_received { term: 82 index: 86 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:10.366532 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 83 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:10.366653 20913 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 83 FOLLOWER]: Leader election lost for term 83. Reason: could not achieve majority
I20251212 03:12:10.374014 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 83 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 82. Current term is 83. Ops: []
I20251212 03:12:10.374249 20908 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 82.86, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20251212 03:12:10.374353 20908 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 82 LEADER]: Stepping down as leader of term 82
I20251212 03:12:10.374392 20908 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 82 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:10.374449 20908 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 82.86, Last appended by leader: 86, Current term: 82, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:10.374655 20908 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 82 FOLLOWER]: Advancing to term 83
I20251212 03:12:10.447580 20911 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 82 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:10.447683 20911 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 82 FOLLOWER]: Advancing to term 83
I20251212 03:12:10.448359 20911 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 83 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:10.448537 20911 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 83 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:10.448764 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 83 candidate_status { last_received { term: 82 index: 86 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:10.448870 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 83 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 83: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:10.448822 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 83 candidate_status { last_received { term: 82 index: 86 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:10.449545 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 83 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 83.
I20251212 03:12:10.449748 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 83 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:10.449832 20911 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 83 FOLLOWER]: Leader election won for term 83
I20251212 03:12:10.449896 20911 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 83 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:10.449954 20911 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 82.86, Last appended by leader: 86, Current term: 83, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:10.450560 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 82 to 83, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 83 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:10.497134 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:10.511108 20913 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 83 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:10.511214 20913 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 83 FOLLOWER]: Advancing to term 84
I20251212 03:12:10.511926 20913 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 84 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:10.512131 20913 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 84 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:10.512377 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 84 candidate_status { last_received { term: 82 index: 86 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:10.512375 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 84 candidate_status { last_received { term: 82 index: 86 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:10.512486 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 83 FOLLOWER]: Advancing to term 84
I20251212 03:12:10.513062 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 84 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 84.
I20251212 03:12:10.513269 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 84 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:10.513361 20913 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 84 FOLLOWER]: Leader election won for term 84
I20251212 03:12:10.513439 20913 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 84 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:10.513499 20913 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 82.86, Last appended by leader: 86, Current term: 84, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:10.513931 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 83 to 84, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 84 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:10.547324 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 84 LEADER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 83. Current term is 84. Ops: []
I20251212 03:12:10.547660 20911 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20251212 03:12:10.547775 20911 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer e9d27948dced4eba8f8228ddc4529393 log is divergent from this leader: its last log entry 84.87 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 86
I20251212 03:12:10.547807 20895 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 83 LEADER]: Stepping down as leader of term 83
I20251212 03:12:10.547869 20895 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 83 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:10.547940 20895 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 83.87, Last appended by leader: 87, Current term: 83, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:10.548013 20895 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 83 FOLLOWER]: Advancing to term 84
I20251212 03:12:10.609212 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 84 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 82 index: 86. Preceding OpId from leader: term: 84 index: 87. (index mismatch)
I20251212 03:12:10.609545 20913 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20251212 03:12:10.610715 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 84 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 83 index: 87. Preceding OpId from leader: term: 84 index: 87. (term mismatch)
I20251212 03:12:10.610775 20759 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting all ops after (but not including) 86
I20251212 03:12:10.610800 20759 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting uncommitted NO_OP operation due to leader change: 83.87
I20251212 03:12:10.610822 20759 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 84 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:10.610996 20915 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20251212 03:12:10.831387 20911 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 84 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:10.831492 20911 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 84 FOLLOWER]: Advancing to term 85
I20251212 03:12:10.832191 20911 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 85 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:10.832363 20911 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 85 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:10.832597 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 85 candidate_status { last_received { term: 84 index: 87 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:10.832629 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 85 candidate_status { last_received { term: 84 index: 87 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:10.832875 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 85 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:10.832966 20911 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 85 FOLLOWER]: Leader election lost for term 85. Reason: could not achieve majority
I20251212 03:12:10.847201 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 85 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 84. Current term is 85. Ops: []
I20251212 03:12:10.847481 20891 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 84.87, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20251212 03:12:10.847644 20891 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 84 LEADER]: Stepping down as leader of term 84
I20251212 03:12:10.847680 20891 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 84 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:10.847745 20891 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 84.87, Last appended by leader: 87, Current term: 84, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:10.847833 20891 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 84 FOLLOWER]: Advancing to term 85
I20251212 03:12:10.902700 20908 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 84 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:10.902817 20908 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 84 FOLLOWER]: Advancing to term 85
I20251212 03:12:10.903491 20908 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 85 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:10.903676 20908 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 85 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:10.903892 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 85 candidate_status { last_received { term: 84 index: 87 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:10.903903 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 85 candidate_status { last_received { term: 84 index: 87 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:10.904011 20759 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 85 FOLLOWER]: Leader election vote request: Denying vote to candidate d3fe965dfda94fd4bf3f61a5c0452d63 in current term 85: Already voted for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in this term.
I20251212 03:12:10.904532 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 85 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 85.
I20251212 03:12:10.904708 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 85 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:10.904798 20908 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 85 FOLLOWER]: Leader election won for term 85
I20251212 03:12:10.904894 20908 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 85 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:10.904935 20908 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 84.87, Last appended by leader: 87, Current term: 85, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:10.905377 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 84 to 85, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 85 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:10.941401 20911 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 85 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:10.941495 20911 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 85 FOLLOWER]: Advancing to term 86
I20251212 03:12:10.942229 20911 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 86 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:10.942391 20911 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 86 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:10.942616 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 86 candidate_status { last_received { term: 84 index: 87 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:10.942627 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 86 candidate_status { last_received { term: 84 index: 87 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:10.942728 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 85 FOLLOWER]: Advancing to term 86
I20251212 03:12:10.943300 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 86 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 86.
I20251212 03:12:10.943493 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 86 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:10.943580 20911 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 86 FOLLOWER]: Leader election won for term 86
I20251212 03:12:10.943632 20911 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 86 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:10.943796 20911 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 84.87, Last appended by leader: 87, Current term: 86, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:10.944243 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 85 to 86, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 86 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:11.001946 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 86 LEADER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 85. Current term is 86. Ops: []
I20251212 03:12:11.002274 20908 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20251212 03:12:11.002401 20908 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer f894b0d567ab4a2ebdcb416d04ef48e5 log is divergent from this leader: its last log entry 86.88 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 87
I20251212 03:12:11.002429 20908 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 85 LEADER]: Stepping down as leader of term 85
I20251212 03:12:11.002444 20908 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 85 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:11.002506 20908 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 85.88, Last appended by leader: 88, Current term: 85, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:11.002579 20908 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 85 FOLLOWER]: Advancing to term 86
I20251212 03:12:11.026764 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 86 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 85 index: 88. Preceding OpId from leader: term: 86 index: 88. (term mismatch)
I20251212 03:12:11.026852 20682 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting all ops after (but not including) 87
I20251212 03:12:11.026867 20682 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting uncommitted NO_OP operation due to leader change: 85.88
I20251212 03:12:11.026882 20682 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 86 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:11.027168 20911 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20251212 03:12:11.028592 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 86 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 84 index: 87. Preceding OpId from leader: term: 86 index: 88. (index mismatch)
I20251212 03:12:11.028940 20917 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20251212 03:12:11.210613 20891 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 86 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:11.210712 20891 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 86 FOLLOWER]: Advancing to term 87
I20251212 03:12:11.211405 20891 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 87 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:11.211582 20891 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 87 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:11.211820 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 87 candidate_status { last_received { term: 86 index: 88 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:11.211829 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 87 candidate_status { last_received { term: 86 index: 88 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:11.212072 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 87 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:11.212169 20891 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 87 FOLLOWER]: Leader election lost for term 87. Reason: could not achieve majority
I20251212 03:12:11.234516 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 87 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 86. Current term is 87. Ops: []
I20251212 03:12:11.234823 20911 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 86.88, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20251212 03:12:11.234951 20911 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 86 LEADER]: Stepping down as leader of term 86
I20251212 03:12:11.234988 20911 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 86 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:11.235061 20911 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 86.88, Last appended by leader: 88, Current term: 86, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:11.235145 20911 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 86 FOLLOWER]: Advancing to term 87
I20251212 03:12:11.241304 20908 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 86 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:11.241402 20908 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 86 FOLLOWER]: Advancing to term 87
I20251212 03:12:11.242005 20908 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 87 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:11.242165 20908 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 87 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:11.242439 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 87 candidate_status { last_received { term: 86 index: 88 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:11.242439 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 87 candidate_status { last_received { term: 86 index: 88 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:11.242584 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 87 FOLLOWER]: Leader election vote request: Denying vote to candidate d3fe965dfda94fd4bf3f61a5c0452d63 in current term 87: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:11.243105 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 87 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 87.
I20251212 03:12:11.243270 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 87 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:11.243386 20908 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 87 FOLLOWER]: Leader election won for term 87
I20251212 03:12:11.243428 20908 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 87 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:11.243484 20908 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 86.88, Last appended by leader: 88, Current term: 87, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:11.243993 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 86 to 87, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 87 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:11.351565 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 87 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 86 index: 88. Preceding OpId from leader: term: 87 index: 89. (index mismatch)
I20251212 03:12:11.351893 20908 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20251212 03:12:11.352975 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 87 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 86 index: 88. Preceding OpId from leader: term: 87 index: 89. (index mismatch)
I20251212 03:12:11.353044 20891 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 87 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:11.353106 20891 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 87 FOLLOWER]: Advancing to term 88
I20251212 03:12:11.353545 20916 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20251212 03:12:11.353870 20891 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 88 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:11.353960 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 88 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 87. Current term is 88. Ops: [87.89-87.89]
I20251212 03:12:11.354013 20891 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 88 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:11.354125 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 88 candidate_status { last_received { term: 86 index: 88 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:11.354177 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 88 candidate_status { last_received { term: 86 index: 88 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:11.354274 20916 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 86.88, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20251212 03:12:11.354350 20916 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 87 LEADER]: Stepping down as leader of term 87
I20251212 03:12:11.354377 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 88 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:11.354382 20916 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 87 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:11.354485 20891 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 88 FOLLOWER]: Leader election lost for term 88. Reason: could not achieve majority
I20251212 03:12:11.354561 20916 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 87.89, Last appended by leader: 89, Current term: 87, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:11.354661 20916 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 87 FOLLOWER]: Advancing to term 88
I20251212 03:12:11.460677 20911 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 87 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:11.460788 20911 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 87 FOLLOWER]: Advancing to term 88
I20251212 03:12:11.461484 20911 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 88 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:11.461661 20911 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 88 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:11.461910 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 88 candidate_status { last_received { term: 87 index: 89 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:11.461930 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 88 candidate_status { last_received { term: 87 index: 89 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:11.462024 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 88 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 88: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:11.462669 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 88 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 88.
I20251212 03:12:11.463004 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 88 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:11.463120 20911 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 88 FOLLOWER]: Leader election won for term 88
I20251212 03:12:11.463189 20911 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 88 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:11.463248 20911 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 87.89, Last appended by leader: 89, Current term: 88, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:11.463773 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 87 to 88, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 88 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:11.497424 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:11.499922 20891 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 88 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:11.500006 20891 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 88 FOLLOWER]: Advancing to term 89
I20251212 03:12:11.500677 20891 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 89 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:11.500845 20891 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 89 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:11.501083 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 89 candidate_status { last_received { term: 86 index: 88 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:11.501081 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 89 candidate_status { last_received { term: 86 index: 88 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:11.501186 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 88 FOLLOWER]: Advancing to term 89
I20251212 03:12:11.501811 20682 raft_consensus.cc:2410] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 89 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 for term 89 because replica has last-logged OpId of term: 87 index: 89, which is greater than that of the candidate, which has last-logged OpId of term: 86 index: 88.
I20251212 03:12:11.502014 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 89 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:11.502138 20891 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 89 FOLLOWER]: Leader election lost for term 89. Reason: could not achieve majority
I20251212 03:12:11.539947 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 89 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 88. Current term is 89. Ops: []
I20251212 03:12:11.540251 20911 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20251212 03:12:11.540373 20911 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 88 LEADER]: Stepping down as leader of term 88
I20251212 03:12:11.540407 20911 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 88 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:11.540475 20911 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 88.90, Last appended by leader: 90, Current term: 88, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:11.540547 20911 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 88 FOLLOWER]: Advancing to term 89
I20251212 03:12:11.612612 20916 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 89 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:11.612722 20916 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 89 FOLLOWER]: Advancing to term 90
I20251212 03:12:11.613456 20916 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 90 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:11.613626 20916 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 90 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:11.613848 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 90 candidate_status { last_received { term: 87 index: 89 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:11.613927 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 89 FOLLOWER]: Advancing to term 90
I20251212 03:12:11.613852 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 90 candidate_status { last_received { term: 87 index: 89 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:11.613982 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 89 FOLLOWER]: Advancing to term 90
I20251212 03:12:11.614588 20759 raft_consensus.cc:2410] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 90 FOLLOWER]: Leader election vote request: Denying vote to candidate d3fe965dfda94fd4bf3f61a5c0452d63 for term 90 because replica has last-logged OpId of term: 88 index: 90, which is greater than that of the candidate, which has last-logged OpId of term: 87 index: 89.
I20251212 03:12:11.614632 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 90 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 90.
I20251212 03:12:11.614812 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 90 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:11.614902 20916 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 90 FOLLOWER]: Leader election won for term 90
I20251212 03:12:11.614969 20916 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 90 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:11.615026 20916 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 87.89, Last appended by leader: 89, Current term: 90, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:11.615552 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 88 to 90, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 90 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:11.699999 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 90 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 88 index: 90. Preceding OpId from leader: term: 90 index: 90. (term mismatch)
I20251212 03:12:11.700074 20759 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting all ops after (but not including) 89
I20251212 03:12:11.700088 20759 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting uncommitted NO_OP operation due to leader change: 88.90
I20251212 03:12:11.700103 20759 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 90 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:11.700312 20916 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20251212 03:12:11.701355 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 90 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 86 index: 88. Preceding OpId from leader: term: 90 index: 90. (index mismatch)
I20251212 03:12:11.701532 20918 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 90, Last known committed idx: 88, Time since last communication: 0.000s
I20251212 03:12:11.885178 20919 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 90 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:11.885284 20919 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 90 FOLLOWER]: Advancing to term 91
I20251212 03:12:11.887001 20919 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 91 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:11.887212 20911 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 90 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:11.887341 20911 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 90 FOLLOWER]: Advancing to term 91
I20251212 03:12:11.887411 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 91 candidate_status { last_received { term: 90 index: 90 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:11.887411 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 91 candidate_status { last_received { term: 90 index: 90 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:11.887516 20919 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 91 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:11.888001 20911 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 91 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:11.888124 20759 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 91 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 in current term 91: Already voted for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in this term.
I20251212 03:12:11.888232 20911 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 91 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:11.888289 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 91 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:11.888338 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 91 candidate_status { last_received { term: 90 index: 90 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:11.888406 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 91 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 91: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:11.888475 20919 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 91 FOLLOWER]: Leader election lost for term 91. Reason: could not achieve majority
I20251212 03:12:11.888392 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 91 candidate_status { last_received { term: 90 index: 90 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:11.888674 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 91 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 90. Current term is 91. Ops: []
I20251212 03:12:11.888707 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 91 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:11.888818 20911 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 91 FOLLOWER]: Leader election lost for term 91. Reason: could not achieve majority
I20251212 03:12:11.888892 20908 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 90.90, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20251212 03:12:11.888991 20908 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 90 LEADER]: Stepping down as leader of term 90
I20251212 03:12:11.889022 20908 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 90 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:11.889086 20908 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 90.90, Last appended by leader: 90, Current term: 90, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:11.889163 20908 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 90 FOLLOWER]: Advancing to term 91
I20251212 03:12:11.996101 20919 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 91 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:11.996217 20919 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 91 FOLLOWER]: Advancing to term 92
I20251212 03:12:11.996922 20919 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 92 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:11.997090 20919 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 92 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:11.997324 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 92 candidate_status { last_received { term: 90 index: 90 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:11.997406 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 91 FOLLOWER]: Advancing to term 92
I20251212 03:12:11.997328 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 92 candidate_status { last_received { term: 90 index: 90 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:11.997455 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 91 FOLLOWER]: Advancing to term 92
I20251212 03:12:11.997984 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 92 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 92.
I20251212 03:12:11.998059 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 92 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 92.
I20251212 03:12:11.998158 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 92 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters:
I20251212 03:12:11.998270 20919 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 92 FOLLOWER]: Leader election won for term 92
I20251212 03:12:11.998328 20919 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 92 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:11.998363 20919 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 90.90, Last appended by leader: 90, Current term: 92, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:11.998782 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 90 to 92, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 92 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:12.080417 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 92 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 90 index: 90. Preceding OpId from leader: term: 92 index: 91. (index mismatch)
I20251212 03:12:12.080708 20919 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20251212 03:12:12.081748 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 92 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 90 index: 90. Preceding OpId from leader: term: 92 index: 91. (index mismatch)
I20251212 03:12:12.082501 20919 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20251212 03:12:12.261051 20908 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 92 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:12.261159 20908 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 92 FOLLOWER]: Advancing to term 93
I20251212 03:12:12.261866 20908 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 93 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:12.262022 20908 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 93 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:12.262253 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 93 candidate_status { last_received { term: 92 index: 91 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:12.262285 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 93 candidate_status { last_received { term: 92 index: 91 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:12.262530 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 93 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:12.262665 20908 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 93 FOLLOWER]: Leader election lost for term 93. Reason: could not achieve majority
I20251212 03:12:12.265632 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 93 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 92. Current term is 93. Ops: []
I20251212 03:12:12.265892 20919 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 92.91, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20251212 03:12:12.266006 20919 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 92 LEADER]: Stepping down as leader of term 92
I20251212 03:12:12.266041 20919 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 92 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:12.266096 20919 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 92.91, Last appended by leader: 91, Current term: 92, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:12.266175 20919 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 92 FOLLOWER]: Advancing to term 93
I20251212 03:12:12.271952 20911 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 92 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:12.272022 20911 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 92 FOLLOWER]: Advancing to term 93
I20251212 03:12:12.272802 20911 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 93 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:12.272976 20911 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 93 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:12.273191 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 93 candidate_status { last_received { term: 92 index: 91 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:12.273195 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 93 candidate_status { last_received { term: 92 index: 91 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:12.273290 20682 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 93 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 93: Already voted for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in this term.
I20251212 03:12:12.273828 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 93 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 93.
I20251212 03:12:12.274035 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 93 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:12.274163 20911 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 93 FOLLOWER]: Leader election won for term 93
I20251212 03:12:12.274379 20911 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 93 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:12.274443 20911 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 92.91, Last appended by leader: 91, Current term: 93, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:12.274953 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 92 to 93, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 93 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:12.367429 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 93 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 92 index: 91. Preceding OpId from leader: term: 93 index: 92. (index mismatch)
I20251212 03:12:12.367726 20911 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20251212 03:12:12.368928 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 93 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 92 index: 91. Preceding OpId from leader: term: 93 index: 92. (index mismatch)
I20251212 03:12:12.369202 20923 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20251212 03:12:12.497704 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:12.553682 20908 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 93 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:12.553717 20919 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 93 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:12.553791 20908 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 93 FOLLOWER]: Advancing to term 94
I20251212 03:12:12.553805 20919 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 93 FOLLOWER]: Advancing to term 94
I20251212 03:12:12.554528 20919 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 94 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:12.554536 20908 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 94 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:12.554704 20919 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 94 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:12.554749 20908 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 94 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:12.554965 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 94 candidate_status { last_received { term: 93 index: 92 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:12.554962 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 94 candidate_status { last_received { term: 93 index: 92 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:12.554960 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 94 candidate_status { last_received { term: 93 index: 92 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:12.555117 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 94 FOLLOWER]: Leader election vote request: Denying vote to candidate d3fe965dfda94fd4bf3f61a5c0452d63 in current term 94: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:12.554983 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 94 candidate_status { last_received { term: 93 index: 92 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:12.555254 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 94 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:12.555285 20682 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 94 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 in current term 94: Already voted for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in this term.
I20251212 03:12:12.555351 20908 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 94 FOLLOWER]: Leader election lost for term 94. Reason: could not achieve majority
I20251212 03:12:12.555454 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 94 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:12.555536 20919 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 94 FOLLOWER]: Leader election lost for term 94. Reason: could not achieve majority
I20251212 03:12:12.566622 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 94 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 93. Current term is 94. Ops: []
I20251212 03:12:12.566917 20921 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 93.92, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20251212 03:12:12.567058 20911 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 93 LEADER]: Stepping down as leader of term 93
I20251212 03:12:12.567144 20911 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 93 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:12.567198 20911 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 93.92, Last appended by leader: 92, Current term: 93, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:12.567281 20911 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 93 FOLLOWER]: Advancing to term 94
I20251212 03:12:12.667025 20908 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 94 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:12.667131 20908 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 94 FOLLOWER]: Advancing to term 95
I20251212 03:12:12.667876 20908 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 95 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:12.668030 20908 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 95 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:12.668279 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 95 candidate_status { last_received { term: 93 index: 92 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:12.668388 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 94 FOLLOWER]: Advancing to term 95
I20251212 03:12:12.668287 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 95 candidate_status { last_received { term: 93 index: 92 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:12.668438 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 94 FOLLOWER]: Advancing to term 95
I20251212 03:12:12.668939 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 95 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 95.
I20251212 03:12:12.669040 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 95 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 95.
I20251212 03:12:12.669147 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 95 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters:
I20251212 03:12:12.669252 20908 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 95 FOLLOWER]: Leader election won for term 95
I20251212 03:12:12.669373 20908 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 95 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:12.669432 20908 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 93.92, Last appended by leader: 92, Current term: 95, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:12.669901 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 93 to 95, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 95 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:12.748468 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 95 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 93 index: 92. Preceding OpId from leader: term: 95 index: 93. (index mismatch)
I20251212 03:12:12.748750 20908 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20251212 03:12:12.749801 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 95 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 93 index: 92. Preceding OpId from leader: term: 95 index: 93. (index mismatch)
I20251212 03:12:12.750028 20908 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20251212 03:12:12.943204 20919 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 95 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:12.943289 20919 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 95 FOLLOWER]: Advancing to term 96
I20251212 03:12:12.943930 20919 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 96 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:12.944069 20919 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 96 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:12.944233 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 96 candidate_status { last_received { term: 95 index: 93 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:12.944280 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 96 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 95. Current term is 96. Ops: []
I20251212 03:12:12.944259 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 96 candidate_status { last_received { term: 95 index: 93 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:12.944504 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 96 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:12.944553 20924 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 95.93, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20251212 03:12:12.944607 20919 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 96 FOLLOWER]: Leader election lost for term 96. Reason: could not achieve majority
I20251212 03:12:12.944676 20925 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 95 LEADER]: Stepping down as leader of term 95
I20251212 03:12:12.944718 20925 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 95 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:12.944787 20925 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 95.93, Last appended by leader: 93, Current term: 95, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:12.944866 20925 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 95 FOLLOWER]: Advancing to term 96
I20251212 03:12:13.042790 20911 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 95 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:13.042897 20911 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 95 FOLLOWER]: Advancing to term 96
I20251212 03:12:13.043583 20911 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 96 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:13.043759 20911 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 96 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:13.043969 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 96 candidate_status { last_received { term: 95 index: 93 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:13.044046 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 96 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 96: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:13.043982 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 96 candidate_status { last_received { term: 95 index: 93 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:13.044644 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 96 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 96.
I20251212 03:12:13.044837 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 96 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:13.044929 20911 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 96 FOLLOWER]: Leader election won for term 96
I20251212 03:12:13.044979 20911 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 96 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:13.045030 20911 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 95.93, Last appended by leader: 93, Current term: 96, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:13.045533 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 95 to 96, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 96 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:13.088099 20919 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 96 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:13.088208 20919 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 96 FOLLOWER]: Advancing to term 97
I20251212 03:12:13.088862 20919 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 97 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:13.089022 20919 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 97 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:13.089262 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 97 candidate_status { last_received { term: 95 index: 93 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:13.089262 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 97 candidate_status { last_received { term: 95 index: 93 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:13.089371 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 96 FOLLOWER]: Advancing to term 97
I20251212 03:12:13.089905 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 97 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 97.
I20251212 03:12:13.090093 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 97 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:13.090184 20919 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 97 FOLLOWER]: Leader election won for term 97
I20251212 03:12:13.090317 20919 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 97 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:13.090364 20919 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 95.93, Last appended by leader: 93, Current term: 97, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:13.090902 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 96 to 97, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 97 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:13.122010 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 97 LEADER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 96. Current term is 97. Ops: []
I20251212 03:12:13.122406 20911 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20251212 03:12:13.122540 20911 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer e9d27948dced4eba8f8228ddc4529393 log is divergent from this leader: its last log entry 97.94 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 93
I20251212 03:12:13.122604 20921 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 96 LEADER]: Stepping down as leader of term 96
I20251212 03:12:13.122699 20921 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 96 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:13.122783 20921 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 96.94, Last appended by leader: 94, Current term: 96, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:13.122874 20921 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 96 FOLLOWER]: Advancing to term 97
I20251212 03:12:13.170334 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 97 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 96 index: 94. Preceding OpId from leader: term: 97 index: 94. (term mismatch)
I20251212 03:12:13.170409 20759 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting all ops after (but not including) 93
I20251212 03:12:13.170423 20759 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting uncommitted NO_OP operation due to leader change: 96.94
I20251212 03:12:13.170442 20759 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 97 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:13.170712 20919 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20251212 03:12:13.171769 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 97 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 95 index: 93. Preceding OpId from leader: term: 97 index: 94. (index mismatch)
I20251212 03:12:13.172003 20926 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20251212 03:12:13.272850 20925 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 97 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:13.272957 20925 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 97 FOLLOWER]: Advancing to term 98
I20251212 03:12:13.273609 20925 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 98 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:13.273777 20925 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 98 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:13.273977 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 98 candidate_status { last_received { term: 97 index: 94 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:13.273988 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 98 candidate_status { last_received { term: 97 index: 94 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:13.274044 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 97 FOLLOWER]: Advancing to term 98
I20251212 03:12:13.274847 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 98 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 98.
I20251212 03:12:13.274937 20758 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 98 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 97. Current term is 98. Ops: []
I20251212 03:12:13.275043 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 98 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:13.275136 20925 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 98 FOLLOWER]: Leader election won for term 98
I20251212 03:12:13.275197 20925 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 98 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:13.275170 20926 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 97.94, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20251212 03:12:13.275285 20919 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 97 LEADER]: Stepping down as leader of term 97
I20251212 03:12:13.275310 20919 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 97 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:13.275384 20919 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 97.94, Last appended by leader: 94, Current term: 97, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:13.275445 20919 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 97 FOLLOWER]: Advancing to term 98
I20251212 03:12:13.275244 20925 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 97.94, Last appended by leader: 94, Current term: 98, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:13.276451 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 97 to 98, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 98 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:13.357710 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 98 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 97 index: 94. Preceding OpId from leader: term: 98 index: 95. (index mismatch)
I20251212 03:12:13.357992 20925 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20251212 03:12:13.358929 20758 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 98 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 97 index: 94. Preceding OpId from leader: term: 98 index: 95. (index mismatch)
I20251212 03:12:13.359169 20925 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20251212 03:12:13.497977 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:13.540740 20911 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 98 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:13.540840 20911 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 98 FOLLOWER]: Advancing to term 99
I20251212 03:12:13.541529 20911 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 99 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:13.541690 20911 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 99 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:13.541906 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 99 candidate_status { last_received { term: 98 index: 95 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:13.541927 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 99 candidate_status { last_received { term: 98 index: 95 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:13.542136 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 99 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:13.542227 20911 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 99 FOLLOWER]: Leader election lost for term 99. Reason: could not achieve majority
I20251212 03:12:13.544651 20758 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 99 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 98. Current term is 99. Ops: []
I20251212 03:12:13.544909 20925 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 98.95, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20251212 03:12:13.545020 20925 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 98 LEADER]: Stepping down as leader of term 98
I20251212 03:12:13.545058 20925 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 98 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:13.545107 20925 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 98.95, Last appended by leader: 95, Current term: 98, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:13.545186 20925 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 98 FOLLOWER]: Advancing to term 99
I20251212 03:12:13.644629 20919 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 98 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:13.644738 20919 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 98 FOLLOWER]: Advancing to term 99
I20251212 03:12:13.645442 20919 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 99 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:13.645642 20919 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 99 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:13.645859 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 99 candidate_status { last_received { term: 98 index: 95 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:13.645859 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 99 candidate_status { last_received { term: 98 index: 95 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:13.645993 20758 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 99 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 in current term 99: Already voted for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in this term.
I20251212 03:12:13.646589 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 99 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 99.
I20251212 03:12:13.646831 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 99 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:13.646935 20919 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 99 FOLLOWER]: Leader election won for term 99
I20251212 03:12:13.646983 20919 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 99 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:13.647053 20919 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 98.95, Last appended by leader: 95, Current term: 99, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:13.647553 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 98 to 99, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 99 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:13.659612 20911 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 99 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:13.659696 20911 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 99 FOLLOWER]: Advancing to term 100
I20251212 03:12:13.660310 20911 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 100 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:13.660468 20911 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 100 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:13.660682 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 100 candidate_status { last_received { term: 98 index: 95 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:13.660755 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 99 FOLLOWER]: Advancing to term 100
I20251212 03:12:13.660687 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 100 candidate_status { last_received { term: 98 index: 95 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:13.661321 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 100 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 100.
I20251212 03:12:13.661501 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 100 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:13.661607 20911 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 100 FOLLOWER]: Leader election won for term 100
I20251212 03:12:13.661676 20911 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 100 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:13.661717 20911 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 98.95, Last appended by leader: 95, Current term: 100, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:13.662163 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 99 to 100, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 100 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:13.743911 20605 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 99 LEADER]: Stepping down as leader of term 99
I20251212 03:12:13.743993 20605 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 99 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:13.744050 20605 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 99.96, Last appended by leader: 96, Current term: 99, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:13.744144 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 99 FOLLOWER]: Advancing to term 100
I20251212 03:12:13.744798 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 100 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 99 index: 96. Preceding OpId from leader: term: 100 index: 96. (term mismatch)
I20251212 03:12:13.744843 20605 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting all ops after (but not including) 95
I20251212 03:12:13.744856 20605 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting uncommitted NO_OP operation due to leader change: 99.96
I20251212 03:12:13.744870 20605 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 100 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:13.745122 20911 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20251212 03:12:13.746254 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 100 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 98 index: 95. Preceding OpId from leader: term: 100 index: 96. (index mismatch)
I20251212 03:12:13.746671 20921 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20251212 03:12:14.020491 20919 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 100 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:14.020622 20919 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 100 FOLLOWER]: Advancing to term 101
I20251212 03:12:14.021318 20919 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 101 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:14.021512 20919 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 101 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:14.021785 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 101 candidate_status { last_received { term: 100 index: 96 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:14.021785 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 101 candidate_status { last_received { term: 100 index: 96 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:14.022087 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 101 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:14.022186 20919 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 101 FOLLOWER]: Leader election lost for term 101. Reason: could not achieve majority
I20251212 03:12:14.028630 20925 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 100 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:14.028712 20925 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 100 FOLLOWER]: Advancing to term 101
I20251212 03:12:14.029410 20925 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 101 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:14.029569 20925 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 101 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:14.029783 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 101 candidate_status { last_received { term: 100 index: 96 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:14.029783 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 101 candidate_status { last_received { term: 100 index: 96 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:14.029915 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 101 FOLLOWER]: Leader election vote request: Denying vote to candidate d3fe965dfda94fd4bf3f61a5c0452d63 in current term 101: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:14.030066 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 101 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:14.030170 20925 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 101 FOLLOWER]: Leader election lost for term 101. Reason: could not achieve majority
I20251212 03:12:14.037972 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 101 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 100. Current term is 101. Ops: []
I20251212 03:12:14.038316 20911 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 100.96, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20251212 03:12:14.038453 20921 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 100 LEADER]: Stepping down as leader of term 100
I20251212 03:12:14.038508 20921 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 100 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:14.038569 20921 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 100.96, Last appended by leader: 96, Current term: 100, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:14.038673 20921 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 100 FOLLOWER]: Advancing to term 101
I20251212 03:12:14.147540 20919 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 101 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:14.147645 20919 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 101 FOLLOWER]: Advancing to term 102
I20251212 03:12:14.148262 20919 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 102 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:14.148419 20919 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 102 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:14.148659 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 102 candidate_status { last_received { term: 100 index: 96 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:14.148644 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 102 candidate_status { last_received { term: 100 index: 96 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:14.148772 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 101 FOLLOWER]: Advancing to term 102
I20251212 03:12:14.148788 20758 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 101 FOLLOWER]: Advancing to term 102
I20251212 03:12:14.149296 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 102 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 102.
I20251212 03:12:14.149336 20758 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 102 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 102.
I20251212 03:12:14.149507 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 102 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters:
I20251212 03:12:14.149605 20919 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 102 FOLLOWER]: Leader election won for term 102
I20251212 03:12:14.149721 20919 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 102 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:14.149775 20919 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 100.96, Last appended by leader: 96, Current term: 102, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:14.150187 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 100 to 102, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 102 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:14.245810 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 102 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 100 index: 96. Preceding OpId from leader: term: 102 index: 97. (index mismatch)
I20251212 03:12:14.246105 20919 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20251212 03:12:14.247280 20758 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 102 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 100 index: 96. Preceding OpId from leader: term: 102 index: 97. (index mismatch)
I20251212 03:12:14.247722 20929 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20251212 03:12:14.348647 20921 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 102 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:14.348743 20921 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 102 FOLLOWER]: Advancing to term 103
I20251212 03:12:14.349414 20921 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 103 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:14.349674 20921 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 103 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:14.349766 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 103 candidate_status { last_received { term: 102 index: 97 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:14.349766 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 103 candidate_status { last_received { term: 102 index: 97 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:14.350041 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 103 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:14.350175 20921 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 103 FOLLOWER]: Leader election lost for term 103. Reason: could not achieve majority
I20251212 03:12:14.360644 20758 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 103 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 102. Current term is 103. Ops: []
I20251212 03:12:14.360991 20928 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 102.97, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20251212 03:12:14.361116 20928 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 102 LEADER]: Stepping down as leader of term 102
I20251212 03:12:14.361161 20928 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 102 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:14.361217 20928 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 102.97, Last appended by leader: 97, Current term: 102, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:14.361327 20928 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 102 FOLLOWER]: Advancing to term 103
I20251212 03:12:14.424312 20925 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 102 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:14.424413 20925 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 102 FOLLOWER]: Advancing to term 103
I20251212 03:12:14.425063 20925 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 103 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:14.425222 20925 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 103 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:14.425478 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 103 candidate_status { last_received { term: 102 index: 97 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:14.425576 20758 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 103 FOLLOWER]: Leader election vote request: Denying vote to candidate d3fe965dfda94fd4bf3f61a5c0452d63 in current term 103: Already voted for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in this term.
I20251212 03:12:14.425580 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 103 candidate_status { last_received { term: 102 index: 97 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:14.426255 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 103 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 103.
I20251212 03:12:14.426451 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 103 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:14.426541 20925 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 103 FOLLOWER]: Leader election won for term 103
I20251212 03:12:14.426648 20925 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 103 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:14.426692 20925 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 102.97, Last appended by leader: 97, Current term: 103, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:14.427122 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 102 to 103, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 103 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:14.486645 20921 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 103 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:14.486750 20921 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 103 FOLLOWER]: Advancing to term 104
I20251212 03:12:14.487430 20921 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 104 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:14.487594 20921 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 104 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:14.487808 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 104 candidate_status { last_received { term: 102 index: 97 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:14.487809 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 104 candidate_status { last_received { term: 102 index: 97 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:14.487922 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 103 FOLLOWER]: Advancing to term 104
I20251212 03:12:14.488492 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 104 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 104.
I20251212 03:12:14.488682 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 104 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:14.488771 20921 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 104 FOLLOWER]: Leader election won for term 104
I20251212 03:12:14.488829 20921 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 104 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:14.488883 20921 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 102.97, Last appended by leader: 97, Current term: 104, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:14.489321 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 103 to 104, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 104 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:14.498278 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:14.502321 20758 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 104 LEADER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 103. Current term is 104. Ops: []
I20251212 03:12:14.502432 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 104 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 103. Current term is 104. Ops: []
I20251212 03:12:14.502526 20925 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20251212 03:12:14.502683 20925 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer f894b0d567ab4a2ebdcb416d04ef48e5 log is divergent from this leader: its last log entry 104.98 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 97
I20251212 03:12:14.502736 20931 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 103 LEADER]: Stepping down as leader of term 103
I20251212 03:12:14.502785 20931 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 103 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:14.502753 20930 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20251212 03:12:14.502875 20931 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 103.98, Last appended by leader: 98, Current term: 103, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:14.502972 20931 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 103 FOLLOWER]: Advancing to term 104
I20251212 03:12:14.568100 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 104 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 103 index: 98. Preceding OpId from leader: term: 104 index: 98. (term mismatch)
I20251212 03:12:14.568167 20682 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting all ops after (but not including) 97
I20251212 03:12:14.568181 20682 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting uncommitted NO_OP operation due to leader change: 103.98
I20251212 03:12:14.568195 20682 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 104 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:14.568432 20921 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20251212 03:12:14.569386 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 104 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 102 index: 97. Preceding OpId from leader: term: 104 index: 98. (index mismatch)
I20251212 03:12:14.569583 20921 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20251212 03:12:14.759102 20928 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 104 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:14.759212 20928 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 104 FOLLOWER]: Advancing to term 105
I20251212 03:12:14.759951 20928 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 105 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:14.760120 20928 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 105 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:14.760341 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 105 candidate_status { last_received { term: 104 index: 98 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:14.760357 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 105 candidate_status { last_received { term: 104 index: 98 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:14.760558 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 105 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:14.760654 20928 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 105 FOLLOWER]: Leader election lost for term 105. Reason: could not achieve majority
I20251212 03:12:14.773545 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 105 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 104. Current term is 105. Ops: []
I20251212 03:12:14.773804 20911 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 104.98, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20251212 03:12:14.773926 20911 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 104 LEADER]: Stepping down as leader of term 104
I20251212 03:12:14.773960 20911 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 104 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:14.774027 20911 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 104.98, Last appended by leader: 98, Current term: 104, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:14.774099 20911 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 104 FOLLOWER]: Advancing to term 105
I20251212 03:12:14.844786 20931 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 104 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:14.844890 20931 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 104 FOLLOWER]: Advancing to term 105
I20251212 03:12:14.845606 20931 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 105 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:14.845777 20931 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 105 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:14.846015 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 105 candidate_status { last_received { term: 104 index: 98 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:14.846037 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 105 candidate_status { last_received { term: 104 index: 98 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:14.846086 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 105 FOLLOWER]: Leader election vote request: Denying vote to candidate d3fe965dfda94fd4bf3f61a5c0452d63 in current term 105: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:14.846738 20758 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 105 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 105.
I20251212 03:12:14.846925 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 105 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:14.847024 20931 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 105 FOLLOWER]: Leader election won for term 105
I20251212 03:12:14.847096 20931 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 105 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:14.847162 20931 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 104.98, Last appended by leader: 98, Current term: 105, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:14.847679 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 104 to 105, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 105 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:14.861094 20928 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 105 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:14.861198 20928 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 105 FOLLOWER]: Advancing to term 106
I20251212 03:12:14.862097 20928 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 106 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:14.862457 20928 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 106 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:14.862576 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 106 candidate_status { last_received { term: 104 index: 98 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:14.862601 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 106 candidate_status { last_received { term: 104 index: 98 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:14.862689 20758 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 105 FOLLOWER]: Advancing to term 106
I20251212 03:12:14.863505 20758 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 106 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 106.
I20251212 03:12:14.863680 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 106 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:14.864104 20928 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 106 FOLLOWER]: Leader election won for term 106
I20251212 03:12:14.864235 20928 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 106 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:14.864305 20928 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 104.98, Last appended by leader: 98, Current term: 106, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:14.864889 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 105 to 106, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 106 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:14.932572 20758 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 106 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 105. Current term is 106. Ops: []
I20251212 03:12:14.932876 20931 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20251212 03:12:14.932992 20931 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 105 LEADER]: Stepping down as leader of term 105
I20251212 03:12:14.933030 20931 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 105 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:14.933097 20931 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 105.99, Last appended by leader: 99, Current term: 105, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:14.933169 20931 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 105 FOLLOWER]: Advancing to term 106
I20251212 03:12:14.974336 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 106 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 105 index: 99. Preceding OpId from leader: term: 106 index: 99. (term mismatch)
I20251212 03:12:14.974411 20682 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting all ops after (but not including) 98
I20251212 03:12:14.974426 20682 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting uncommitted NO_OP operation due to leader change: 105.99
I20251212 03:12:14.974439 20682 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 106 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:14.974658 20928 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20251212 03:12:14.975651 20758 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 106 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 104 index: 98. Preceding OpId from leader: term: 106 index: 99. (index mismatch)
I20251212 03:12:14.975814 20928 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20251212 03:12:15.076768 20921 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 106 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:15.076854 20921 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 106 FOLLOWER]: Advancing to term 107
I20251212 03:12:15.077540 20921 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 107 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:15.077702 20921 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 107 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:15.077909 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 107 candidate_status { last_received { term: 106 index: 99 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:15.077921 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 107 candidate_status { last_received { term: 106 index: 99 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:15.078254 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 107 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:15.078348 20921 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 107 FOLLOWER]: Leader election lost for term 107. Reason: could not achieve majority
I20251212 03:12:15.085155 20758 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 107 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 106. Current term is 107. Ops: []
I20251212 03:12:15.085407 20928 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 106.99, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20251212 03:12:15.085521 20928 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 106 LEADER]: Stepping down as leader of term 106
I20251212 03:12:15.085556 20928 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 106 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:15.085614 20928 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 106.99, Last appended by leader: 99, Current term: 106, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:15.085693 20928 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 106 FOLLOWER]: Advancing to term 107
I20251212 03:12:15.176596 20931 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 106 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:15.176699 20931 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 106 FOLLOWER]: Advancing to term 107
I20251212 03:12:15.177382 20931 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 107 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:15.177558 20931 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 107 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:15.177790 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 107 candidate_status { last_received { term: 106 index: 99 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:15.177790 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 107 candidate_status { last_received { term: 106 index: 99 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:15.177924 20758 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 107 FOLLOWER]: Leader election vote request: Denying vote to candidate d3fe965dfda94fd4bf3f61a5c0452d63 in current term 107: Already voted for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in this term.
I20251212 03:12:15.178498 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 107 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 107.
I20251212 03:12:15.178680 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 107 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:15.178766 20931 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 107 FOLLOWER]: Leader election won for term 107
I20251212 03:12:15.178831 20931 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 107 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:15.178895 20931 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 106.99, Last appended by leader: 99, Current term: 107, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:15.179358 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 106 to 107, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 107 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:15.201776 20921 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 107 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:15.201882 20921 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 107 FOLLOWER]: Advancing to term 108
I20251212 03:12:15.202731 20921 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 108 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:15.202893 20921 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 108 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:15.203094 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 108 candidate_status { last_received { term: 106 index: 99 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:15.203123 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 108 candidate_status { last_received { term: 106 index: 99 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:15.203171 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 107 FOLLOWER]: Advancing to term 108
I20251212 03:12:15.203729 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 108 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 108.
I20251212 03:12:15.203934 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 108 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:15.204025 20921 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 108 FOLLOWER]: Leader election won for term 108
I20251212 03:12:15.204101 20921 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 108 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:15.204162 20921 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 106.99, Last appended by leader: 99, Current term: 108, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:15.204618 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 107 to 108, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 108 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:15.269730 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 108 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 107. Current term is 108. Ops: []
I20251212 03:12:15.270004 20931 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20251212 03:12:15.270131 20931 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 107 LEADER]: Stepping down as leader of term 107
I20251212 03:12:15.270174 20931 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 107 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:15.270262 20931 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 107.100, Last appended by leader: 100, Current term: 107, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:15.270344 20931 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 107 FOLLOWER]: Advancing to term 108
I20251212 03:12:15.316069 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 108 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 106 index: 99. Preceding OpId from leader: term: 108 index: 100. (index mismatch)
I20251212 03:12:15.316380 20921 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20251212 03:12:15.317548 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 108 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 107 index: 100. Preceding OpId from leader: term: 108 index: 100. (term mismatch)
I20251212 03:12:15.317605 20682 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting all ops after (but not including) 99
I20251212 03:12:15.317625 20682 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting uncommitted NO_OP operation due to leader change: 107.100
I20251212 03:12:15.317647 20682 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 108 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:15.317821 20921 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20251212 03:12:15.498514 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:15.502969 20928 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 108 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:15.503047 20928 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 108 FOLLOWER]: Advancing to term 109
I20251212 03:12:15.503700 20928 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 109 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:15.503849 20928 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 109 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:15.504042 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 109 candidate_status { last_received { term: 108 index: 100 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:15.504045 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 109 candidate_status { last_received { term: 108 index: 100 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:15.504287 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 109 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:15.504375 20928 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 109 FOLLOWER]: Leader election lost for term 109. Reason: could not achieve majority
I20251212 03:12:15.513408 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 109 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 108. Current term is 109. Ops: []
I20251212 03:12:15.513737 20921 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 108.100, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20251212 03:12:15.513882 20911 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 108 LEADER]: Stepping down as leader of term 108
I20251212 03:12:15.513926 20911 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 108 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:15.513984 20911 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 108.100, Last appended by leader: 100, Current term: 108, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:15.514067 20911 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 108 FOLLOWER]: Advancing to term 109
I20251212 03:12:15.601186 20931 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 108 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:15.601287 20931 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 108 FOLLOWER]: Advancing to term 109
I20251212 03:12:15.601965 20931 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 109 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:15.602118 20931 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 109 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:15.602368 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 109 candidate_status { last_received { term: 108 index: 100 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:15.602447 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 109 FOLLOWER]: Leader election vote request: Denying vote to candidate d3fe965dfda94fd4bf3f61a5c0452d63 in current term 109: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:15.602373 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 109 candidate_status { last_received { term: 108 index: 100 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:15.603041 20758 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 109 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 109.
I20251212 03:12:15.603214 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 109 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:15.603300 20931 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 109 FOLLOWER]: Leader election won for term 109
I20251212 03:12:15.603363 20931 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 109 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:15.603422 20931 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 108.100, Last appended by leader: 100, Current term: 109, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:15.603878 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 108 to 109, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 109 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:15.647925 20928 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 109 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:15.648031 20928 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 109 FOLLOWER]: Advancing to term 110
I20251212 03:12:15.648723 20928 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 110 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:15.648902 20928 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 110 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:15.649134 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 110 candidate_status { last_received { term: 108 index: 100 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:15.649133 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 110 candidate_status { last_received { term: 108 index: 100 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:15.649230 20758 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 109 FOLLOWER]: Advancing to term 110
I20251212 03:12:15.649798 20758 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 110 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 110.
I20251212 03:12:15.649969 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 110 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:15.650055 20928 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 110 FOLLOWER]: Leader election won for term 110
I20251212 03:12:15.650177 20928 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 110 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:15.650271 20928 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 108.100, Last appended by leader: 100, Current term: 110, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:15.650705 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 109 to 110, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 110 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:15.716483 20758 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 110 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 109. Current term is 110. Ops: []
I20251212 03:12:15.716792 20931 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20251212 03:12:15.716919 20931 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 109 LEADER]: Stepping down as leader of term 109
I20251212 03:12:15.716956 20931 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 109 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:15.717021 20931 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 109.101, Last appended by leader: 101, Current term: 109, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:15.717108 20931 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 109 FOLLOWER]: Advancing to term 110
I20251212 03:12:15.727996 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 110 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 109 index: 101. Preceding OpId from leader: term: 110 index: 101. (term mismatch)
I20251212 03:12:15.728088 20682 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting all ops after (but not including) 100
I20251212 03:12:15.728108 20682 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting uncommitted NO_OP operation due to leader change: 109.101
I20251212 03:12:15.728129 20682 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 110 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:15.728322 20928 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20251212 03:12:15.729396 20758 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 110 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 108 index: 100. Preceding OpId from leader: term: 110 index: 101. (index mismatch)
I20251212 03:12:15.729621 20935 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20251212 03:12:15.830227 20911 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 110 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:15.830336 20911 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 110 FOLLOWER]: Advancing to term 111
I20251212 03:12:15.831023 20911 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 111 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:15.831192 20911 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 111 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:15.831430 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 111 candidate_status { last_received { term: 110 index: 101 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:15.831476 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 111 candidate_status { last_received { term: 110 index: 101 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:15.831596 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 110 FOLLOWER]: Advancing to term 111
I20251212 03:12:15.832161 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 111 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 111.
I20251212 03:12:15.832432 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 111 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:15.832553 20911 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 111 FOLLOWER]: Leader election won for term 111
I20251212 03:12:15.832629 20911 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 111 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:15.832695 20911 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 110.101, Last appended by leader: 101, Current term: 111, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:15.833132 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 110 to 111, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 111 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:15.844511 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 111 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 110. Current term is 111. Ops: []
I20251212 03:12:15.844786 20935 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 110.101, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20251212 03:12:15.844900 20935 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 110 LEADER]: Stepping down as leader of term 110
I20251212 03:12:15.844941 20935 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 110 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:15.845016 20935 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 110.101, Last appended by leader: 101, Current term: 110, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:15.845098 20935 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 110 FOLLOWER]: Advancing to term 111
I20251212 03:12:15.935632 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 111 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 110 index: 101. Preceding OpId from leader: term: 111 index: 102. (index mismatch)
I20251212 03:12:15.935894 20911 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20251212 03:12:15.936846 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 111 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 110 index: 101. Preceding OpId from leader: term: 111 index: 102. (index mismatch)
I20251212 03:12:15.937275 20936 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20251212 03:12:16.116617 20935 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 111 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:16.116712 20935 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 111 FOLLOWER]: Advancing to term 112
I20251212 03:12:16.117409 20935 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 112 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:16.117573 20935 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 112 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:16.117762 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 112 candidate_status { last_received { term: 111 index: 102 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:16.117771 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 112 candidate_status { last_received { term: 111 index: 102 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:16.118006 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 112 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:16.118099 20935 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 112 FOLLOWER]: Leader election lost for term 112. Reason: could not achieve majority
I20251212 03:12:16.132508 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 112 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 111. Current term is 112. Ops: []
I20251212 03:12:16.132752 20936 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 111.102, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20251212 03:12:16.132876 20936 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 111 LEADER]: Stepping down as leader of term 111
I20251212 03:12:16.132920 20936 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 111 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:16.132979 20936 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 111.102, Last appended by leader: 102, Current term: 111, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:16.133056 20936 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 111 FOLLOWER]: Advancing to term 112
I20251212 03:12:16.214847 20925 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 111 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:16.214952 20925 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 111 FOLLOWER]: Advancing to term 112
I20251212 03:12:16.215651 20925 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 112 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:16.215821 20925 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 112 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:16.216040 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 112 candidate_status { last_received { term: 111 index: 102 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:16.216059 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 112 candidate_status { last_received { term: 111 index: 102 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:16.216161 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 112 FOLLOWER]: Leader election vote request: Denying vote to candidate d3fe965dfda94fd4bf3f61a5c0452d63 in current term 112: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:16.216720 20758 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 112 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 112.
I20251212 03:12:16.216892 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 112 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:16.216990 20925 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 112 FOLLOWER]: Leader election won for term 112
I20251212 03:12:16.217042 20925 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 112 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:16.217101 20925 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 111.102, Last appended by leader: 102, Current term: 112, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:16.217547 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 111 to 112, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 112 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:16.242655 20935 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 112 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:16.242753 20935 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 112 FOLLOWER]: Advancing to term 113
I20251212 03:12:16.243398 20935 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 113 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:16.243546 20935 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 113 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:16.243791 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 113 candidate_status { last_received { term: 111 index: 102 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:16.243805 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 113 candidate_status { last_received { term: 111 index: 102 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:16.243880 20758 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 112 FOLLOWER]: Advancing to term 113
I20251212 03:12:16.244451 20758 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 113 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 113.
I20251212 03:12:16.244664 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 113 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:16.244757 20935 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 113 FOLLOWER]: Leader election won for term 113
I20251212 03:12:16.244809 20935 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 113 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:16.244870 20935 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 111.102, Last appended by leader: 102, Current term: 113, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:16.245307 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 112 to 113, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 113 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:16.319952 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 113 LEADER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 112. Current term is 113. Ops: []
I20251212 03:12:16.320237 20925 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20251212 03:12:16.320367 20925 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer e9d27948dced4eba8f8228ddc4529393 log is divergent from this leader: its last log entry 113.103 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 102
I20251212 03:12:16.320406 20931 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 112 LEADER]: Stepping down as leader of term 112
I20251212 03:12:16.320448 20931 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 112 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:16.320501 20931 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 112.103, Last appended by leader: 103, Current term: 112, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:16.320585 20931 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 112 FOLLOWER]: Advancing to term 113
I20251212 03:12:16.341012 20758 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 113 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 111 index: 102. Preceding OpId from leader: term: 113 index: 103. (index mismatch)
I20251212 03:12:16.341343 20935 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20251212 03:12:16.342372 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 113 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 112 index: 103. Preceding OpId from leader: term: 113 index: 103. (term mismatch)
I20251212 03:12:16.342437 20682 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting all ops after (but not including) 102
I20251212 03:12:16.342463 20682 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting uncommitted NO_OP operation due to leader change: 112.103
I20251212 03:12:16.342485 20682 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 113 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:16.342655 20934 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20251212 03:12:16.498761 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:16.680176 20925 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 113 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:16.680279 20925 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 113 FOLLOWER]: Advancing to term 114
I20251212 03:12:16.680986 20925 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 114 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:16.681143 20925 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 114 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:16.681420 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 114 candidate_status { last_received { term: 113 index: 103 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:16.681420 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 114 candidate_status { last_received { term: 113 index: 103 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:16.681696 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 114 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:16.681783 20925 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 114 FOLLOWER]: Leader election lost for term 114. Reason: could not achieve majority
I20251212 03:12:16.685930 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 114 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 113. Current term is 114. Ops: []
I20251212 03:12:16.686177 20935 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 113.103, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20251212 03:12:16.686347 20935 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 113 LEADER]: Stepping down as leader of term 113
I20251212 03:12:16.686386 20935 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 113 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:16.686455 20935 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 113.103, Last appended by leader: 103, Current term: 113, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:16.686545 20935 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 113 FOLLOWER]: Advancing to term 114
I20251212 03:12:16.709592 20936 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 113 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:16.709704 20936 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 113 FOLLOWER]: Advancing to term 114
I20251212 03:12:16.710554 20936 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 114 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:16.710757 20936 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 114 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:16.710999 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 114 candidate_status { last_received { term: 113 index: 103 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:16.710999 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 114 candidate_status { last_received { term: 113 index: 103 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:16.711133 20682 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 114 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 114: Already voted for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in this term.
I20251212 03:12:16.711767 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 114 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 114.
I20251212 03:12:16.711958 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 114 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:16.712049 20936 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 114 FOLLOWER]: Leader election won for term 114
I20251212 03:12:16.712152 20936 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 114 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:16.712198 20936 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 113.103, Last appended by leader: 103, Current term: 114, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:16.712682 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 113 to 114, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 114 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:16.796203 20925 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 114 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:16.796329 20925 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 114 FOLLOWER]: Advancing to term 115
I20251212 03:12:16.797022 20925 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 115 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:16.797207 20925 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 115 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:16.797449 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 115 candidate_status { last_received { term: 113 index: 103 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:16.797449 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 115 candidate_status { last_received { term: 113 index: 103 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:16.797580 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 114 FOLLOWER]: Advancing to term 115
I20251212 03:12:16.798141 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 115 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 115.
I20251212 03:12:16.798378 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 115 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:16.798467 20925 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 115 FOLLOWER]: Leader election won for term 115
I20251212 03:12:16.798525 20925 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 115 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:16.798573 20925 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 113.103, Last appended by leader: 103, Current term: 115, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:16.799046 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 114 to 115, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 115 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:16.816118 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 115 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 114. Current term is 115. Ops: []
I20251212 03:12:16.816395 20936 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20251212 03:12:16.816495 20936 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 114 LEADER]: Stepping down as leader of term 114
I20251212 03:12:16.816531 20936 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 114 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:16.816607 20936 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 114.104, Last appended by leader: 104, Current term: 114, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:16.816681 20936 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 114 FOLLOWER]: Advancing to term 115
I20251212 03:12:16.888700 20758 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 115 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 114 index: 104. Preceding OpId from leader: term: 115 index: 104. (term mismatch)
I20251212 03:12:16.888783 20758 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting all ops after (but not including) 103
I20251212 03:12:16.888798 20758 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting uncommitted NO_OP operation due to leader change: 114.104
I20251212 03:12:16.888813 20758 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 115 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:16.889048 20925 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20251212 03:12:16.890273 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 115 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 113 index: 103. Preceding OpId from leader: term: 115 index: 104. (index mismatch)
I20251212 03:12:16.890503 20939 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20251212 03:12:16.991118 20935 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 115 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:16.991236 20935 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 115 FOLLOWER]: Advancing to term 116
I20251212 03:12:16.991983 20935 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 116 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:16.992141 20935 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 116 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:16.992384 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 116 candidate_status { last_received { term: 115 index: 104 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:16.992476 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 116 candidate_status { last_received { term: 115 index: 104 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:16.992729 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 116 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:16.992849 20935 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 116 FOLLOWER]: Leader election lost for term 116. Reason: could not achieve majority
I20251212 03:12:17.009984 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 116 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 115. Current term is 116. Ops: []
I20251212 03:12:17.010462 20939 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 115.104, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20251212 03:12:17.010581 20939 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 115 LEADER]: Stepping down as leader of term 115
I20251212 03:12:17.010622 20939 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 115 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:17.010684 20939 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 115.104, Last appended by leader: 104, Current term: 115, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:17.010762 20939 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 115 FOLLOWER]: Advancing to term 116
I20251212 03:12:17.079093 20936 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 115 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:17.079209 20936 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 115 FOLLOWER]: Advancing to term 116
I20251212 03:12:17.079939 20936 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 116 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:17.080106 20936 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 116 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:17.080360 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 116 candidate_status { last_received { term: 115 index: 104 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:17.080430 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 116 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 116: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:17.080376 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 116 candidate_status { last_received { term: 115 index: 104 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:17.081036 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 116 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 116.
I20251212 03:12:17.081202 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 116 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:17.081284 20936 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 116 FOLLOWER]: Leader election won for term 116
I20251212 03:12:17.081346 20936 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 116 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:17.081401 20936 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 115.104, Last appended by leader: 104, Current term: 116, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:17.081820 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 115 to 116, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 116 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:17.106681 20935 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 116 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:17.106784 20935 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 116 FOLLOWER]: Advancing to term 117
I20251212 03:12:17.107472 20935 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 117 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:17.107652 20935 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 117 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:17.107882 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 117 candidate_status { last_received { term: 115 index: 104 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:17.107899 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 117 candidate_status { last_received { term: 115 index: 104 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:17.107980 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 116 FOLLOWER]: Advancing to term 117
I20251212 03:12:17.108552 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 117 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 117.
I20251212 03:12:17.108732 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 117 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:17.108822 20935 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 117 FOLLOWER]: Leader election won for term 117
I20251212 03:12:17.108872 20935 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 117 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:17.108932 20935 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 115.104, Last appended by leader: 104, Current term: 117, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:17.109436 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 116 to 117, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 117 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:17.170418 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 117 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 116. Current term is 117. Ops: []
I20251212 03:12:17.170833 20936 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20251212 03:12:17.170962 20936 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 116 LEADER]: Stepping down as leader of term 116
I20251212 03:12:17.170981 20936 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 116 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:17.171041 20936 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 116.105, Last appended by leader: 105, Current term: 116, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:17.171128 20936 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 116 FOLLOWER]: Advancing to term 117
I20251212 03:12:17.189208 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 117 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 115 index: 104. Preceding OpId from leader: term: 117 index: 105. (index mismatch)
I20251212 03:12:17.189509 20935 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20251212 03:12:17.190536 20758 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 117 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 116 index: 105. Preceding OpId from leader: term: 117 index: 105. (term mismatch)
I20251212 03:12:17.190598 20758 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting all ops after (but not including) 104
I20251212 03:12:17.190621 20758 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting uncommitted NO_OP operation due to leader change: 116.105
I20251212 03:12:17.190639 20758 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 117 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:17.190800 20937 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20251212 03:12:17.291007 20939 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 117 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:17.291110 20939 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 117 FOLLOWER]: Advancing to term 118
I20251212 03:12:17.291827 20939 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 118 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:17.291980 20939 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 118 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:17.292200 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 118 candidate_status { last_received { term: 117 index: 105 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:17.292214 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 118 candidate_status { last_received { term: 117 index: 105 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:17.292480 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 118 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:17.292642 20939 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 118 FOLLOWER]: Leader election lost for term 118. Reason: could not achieve majority
I20251212 03:12:17.309190 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 118 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 117. Current term is 118. Ops: []
I20251212 03:12:17.309515 20940 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 117.105, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20251212 03:12:17.309656 20935 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 117 LEADER]: Stepping down as leader of term 117
I20251212 03:12:17.309713 20935 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 117 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:17.309804 20935 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 117.105, Last appended by leader: 105, Current term: 117, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:17.309896 20935 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 117 FOLLOWER]: Advancing to term 118
I20251212 03:12:17.396319 20936 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 117 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:17.396435 20936 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 117 FOLLOWER]: Advancing to term 118
I20251212 03:12:17.397506 20936 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 118 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:17.397737 20936 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 118 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:17.397943 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 118 candidate_status { last_received { term: 117 index: 105 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:17.397961 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 118 candidate_status { last_received { term: 117 index: 105 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:17.398042 20682 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 118 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 118: Already voted for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in this term.
I20251212 03:12:17.398854 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 118 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 118.
I20251212 03:12:17.399060 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 118 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:17.399150 20936 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 118 FOLLOWER]: Leader election won for term 118
I20251212 03:12:17.399210 20936 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 118 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:17.399271 20936 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 117.105, Last appended by leader: 105, Current term: 118, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:17.399797 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 117 to 118, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 118 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:17.433458 20939 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 118 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:17.433616 20939 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 118 FOLLOWER]: Advancing to term 119
I20251212 03:12:17.434707 20939 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 119 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:17.435034 20939 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 119 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:17.435122 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 119 candidate_status { last_received { term: 117 index: 105 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:17.435122 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 119 candidate_status { last_received { term: 117 index: 105 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:17.435251 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 118 FOLLOWER]: Advancing to term 119
I20251212 03:12:17.435855 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 119 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 119.
I20251212 03:12:17.436035 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 119 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:17.436147 20939 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 119 FOLLOWER]: Leader election won for term 119
I20251212 03:12:17.436218 20939 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 119 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:17.436296 20939 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 117.105, Last appended by leader: 105, Current term: 119, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:17.436776 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 118 to 119, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 119 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:17.481127 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 119 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 118. Current term is 119. Ops: []
I20251212 03:12:17.481417 20936 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20251212 03:12:17.481535 20936 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 118 LEADER]: Stepping down as leader of term 118
I20251212 03:12:17.481567 20936 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 118 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:17.481647 20936 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 118.106, Last appended by leader: 106, Current term: 118, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:17.481715 20936 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 118 FOLLOWER]: Advancing to term 119
I20251212 03:12:17.504655 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:17.517334 20758 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 119 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 118 index: 106. Preceding OpId from leader: term: 119 index: 106. (term mismatch)
I20251212 03:12:17.517414 20758 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting all ops after (but not including) 105
I20251212 03:12:17.517429 20758 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting uncommitted NO_OP operation due to leader change: 118.106
I20251212 03:12:17.517462 20758 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 119 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:17.517669 20939 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20251212 03:12:17.518774 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 119 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 117 index: 105. Preceding OpId from leader: term: 119 index: 106. (index mismatch)
I20251212 03:12:17.519013 20941 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20251212 03:12:17.620177 20935 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 119 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:17.620270 20935 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 119 FOLLOWER]: Advancing to term 120
I20251212 03:12:17.620960 20935 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 120 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:17.621125 20935 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 120 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:17.621340 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 120 candidate_status { last_received { term: 119 index: 106 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:17.621340 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 120 candidate_status { last_received { term: 119 index: 106 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:17.621608 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 120 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:17.621723 20935 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 120 FOLLOWER]: Leader election lost for term 120. Reason: could not achieve majority
I20251212 03:12:17.633517 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 120 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 119. Current term is 120. Ops: []
I20251212 03:12:17.633752 20939 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 119.106, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20251212 03:12:17.633875 20941 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 119 LEADER]: Stepping down as leader of term 119
I20251212 03:12:17.633913 20941 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 119 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:17.633987 20941 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 119.106, Last appended by leader: 106, Current term: 119, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:17.634081 20941 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 119 FOLLOWER]: Advancing to term 120
I20251212 03:12:17.715873 20936 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 119 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:17.715981 20936 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 119 FOLLOWER]: Advancing to term 120
I20251212 03:12:17.716699 20936 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 120 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:17.716866 20936 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 120 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:17.717093 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 120 candidate_status { last_received { term: 119 index: 106 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:17.717103 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 120 candidate_status { last_received { term: 119 index: 106 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:17.717207 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 120 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 120: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:17.717806 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 120 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 120.
I20251212 03:12:17.718027 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 120 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:17.718127 20936 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 120 FOLLOWER]: Leader election won for term 120
I20251212 03:12:17.718196 20936 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 120 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:17.718259 20936 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 119.106, Last appended by leader: 106, Current term: 120, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:17.718726 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 119 to 120, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 120 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:17.731560 20935 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 120 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:17.731659 20935 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 120 FOLLOWER]: Advancing to term 121
I20251212 03:12:17.732368 20935 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 121 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:17.732518 20935 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 121 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:17.732719 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 121 candidate_status { last_received { term: 119 index: 106 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:17.732725 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 121 candidate_status { last_received { term: 119 index: 106 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:17.732813 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 120 FOLLOWER]: Advancing to term 121
I20251212 03:12:17.733336 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 121 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 121.
I20251212 03:12:17.733510 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 121 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:17.733632 20935 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 121 FOLLOWER]: Leader election won for term 121
I20251212 03:12:17.733690 20935 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 121 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:17.733742 20935 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 119.106, Last appended by leader: 106, Current term: 121, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:17.734191 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 120 to 121, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 121 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:17.801579 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 121 LEADER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 120. Current term is 121. Ops: []
I20251212 03:12:17.801846 20936 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20251212 03:12:17.801954 20936 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer e9d27948dced4eba8f8228ddc4529393 log is divergent from this leader: its last log entry 121.107 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 106
I20251212 03:12:17.801991 20938 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 120 LEADER]: Stepping down as leader of term 120
I20251212 03:12:17.802044 20938 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 120 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:17.802121 20938 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 120.107, Last appended by leader: 107, Current term: 120, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:17.802238 20938 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 120 FOLLOWER]: Advancing to term 121
I20251212 03:12:17.815574 20758 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 121 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 120 index: 107. Preceding OpId from leader: term: 121 index: 107. (term mismatch)
I20251212 03:12:17.815652 20758 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting all ops after (but not including) 106
I20251212 03:12:17.815676 20758 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting uncommitted NO_OP operation due to leader change: 120.107
I20251212 03:12:17.815696 20758 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 121 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:17.815924 20935 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20251212 03:12:17.816987 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 121 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 119 index: 106. Preceding OpId from leader: term: 121 index: 107. (index mismatch)
I20251212 03:12:17.817255 20935 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20251212 03:12:17.917883 20941 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 121 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:17.917980 20941 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 121 FOLLOWER]: Advancing to term 122
I20251212 03:12:17.918722 20941 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 122 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:17.918881 20941 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 122 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:17.919078 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 122 candidate_status { last_received { term: 121 index: 107 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:17.919080 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 122 candidate_status { last_received { term: 121 index: 107 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:17.919385 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 122 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:17.919477 20941 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 122 FOLLOWER]: Leader election lost for term 122. Reason: could not achieve majority
I20251212 03:12:17.926970 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 122 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 121. Current term is 122. Ops: []
I20251212 03:12:17.927291 20942 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 121.107, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20251212 03:12:17.927397 20942 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 121 LEADER]: Stepping down as leader of term 121
I20251212 03:12:17.927426 20942 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 121 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:17.927490 20942 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 121.107, Last appended by leader: 107, Current term: 121, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:17.927582 20942 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 121 FOLLOWER]: Advancing to term 122
I20251212 03:12:18.027668 20938 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 121 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:18.027774 20938 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 121 FOLLOWER]: Advancing to term 122
I20251212 03:12:18.028519 20938 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 122 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:18.028692 20938 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 122 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:18.028916 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 122 candidate_status { last_received { term: 121 index: 107 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:18.028928 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 122 candidate_status { last_received { term: 121 index: 107 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:18.029031 20682 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 122 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 122: Already voted for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in this term.
I20251212 03:12:18.029623 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 122 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 122.
I20251212 03:12:18.029860 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 122 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:18.029958 20938 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 122 FOLLOWER]: Leader election won for term 122
I20251212 03:12:18.030014 20938 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 122 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:18.030058 20938 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 121.107, Last appended by leader: 107, Current term: 122, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:18.030547 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 121 to 122, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 122 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:18.048426 20941 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 122 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:18.048530 20941 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 122 FOLLOWER]: Advancing to term 123
I20251212 03:12:18.049221 20941 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 123 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:18.049430 20941 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 123 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:18.049597 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 123 candidate_status { last_received { term: 121 index: 107 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:18.049650 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 123 candidate_status { last_received { term: 121 index: 107 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:18.049746 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 122 FOLLOWER]: Advancing to term 123
I20251212 03:12:18.050330 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 123 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 123.
I20251212 03:12:18.050518 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 123 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:18.050642 20941 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 123 FOLLOWER]: Leader election won for term 123
I20251212 03:12:18.050725 20941 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 123 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:18.050786 20941 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 121.107, Last appended by leader: 107, Current term: 123, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:18.051290 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 122 to 123, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 123 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:18.107970 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 123 LEADER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 122. Current term is 123. Ops: []
I20251212 03:12:18.108259 20938 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20251212 03:12:18.108390 20938 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer d3fe965dfda94fd4bf3f61a5c0452d63 log is divergent from this leader: its last log entry 123.108 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 107
I20251212 03:12:18.108436 20936 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 122 LEADER]: Stepping down as leader of term 122
I20251212 03:12:18.108491 20936 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 122 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:18.108556 20936 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 122.108, Last appended by leader: 108, Current term: 122, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:18.108629 20936 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 122 FOLLOWER]: Advancing to term 123
I20251212 03:12:18.131803 20758 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 123 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 122 index: 108. Preceding OpId from leader: term: 123 index: 108. (term mismatch)
I20251212 03:12:18.131887 20758 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting all ops after (but not including) 107
I20251212 03:12:18.131918 20758 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting uncommitted NO_OP operation due to leader change: 122.108
I20251212 03:12:18.131939 20758 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 123 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:18.132177 20941 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20251212 03:12:18.133248 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 123 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 121 index: 107. Preceding OpId from leader: term: 123 index: 108. (index mismatch)
I20251212 03:12:18.133412 20939 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20251212 03:12:18.234385 20935 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 123 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:18.234494 20935 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 123 FOLLOWER]: Advancing to term 124
I20251212 03:12:18.235180 20935 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 124 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:18.235352 20935 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 124 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:18.235605 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 124 candidate_status { last_received { term: 123 index: 108 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:18.235615 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 124 candidate_status { last_received { term: 123 index: 108 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:18.235867 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 124 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:18.235991 20935 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 124 FOLLOWER]: Leader election lost for term 124. Reason: could not achieve majority
I20251212 03:12:18.258615 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 124 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 123. Current term is 124. Ops: []
I20251212 03:12:18.258877 20943 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 123.108, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I20251212 03:12:18.259009 20943 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 123 LEADER]: Stepping down as leader of term 123
I20251212 03:12:18.259047 20943 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 123 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:18.259116 20943 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 123.108, Last appended by leader: 108, Current term: 123, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:18.259204 20943 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 123 FOLLOWER]: Advancing to term 124
I20251212 03:12:18.334376 20936 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 123 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:18.334478 20936 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 123 FOLLOWER]: Advancing to term 124
I20251212 03:12:18.335150 20936 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 124 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:18.335330 20936 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 124 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:18.335525 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 124 candidate_status { last_received { term: 123 index: 108 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:18.335536 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 124 candidate_status { last_received { term: 123 index: 108 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:18.335597 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 124 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 124: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:18.336151 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 124 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 124.
I20251212 03:12:18.336336 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 124 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:18.336426 20936 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 124 FOLLOWER]: Leader election won for term 124
I20251212 03:12:18.336478 20936 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 124 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:18.336530 20936 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 123.108, Last appended by leader: 108, Current term: 124, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:18.337049 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 123 to 124, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 124 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:18.358476 20935 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 124 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:18.358561 20935 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 124 FOLLOWER]: Advancing to term 125
I20251212 03:12:18.359189 20935 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 125 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:18.359373 20935 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 125 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:18.359609 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 125 candidate_status { last_received { term: 123 index: 108 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:18.359691 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 124 FOLLOWER]: Advancing to term 125
I20251212 03:12:18.359615 20758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 125 candidate_status { last_received { term: 123 index: 108 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:18.360301 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 125 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 125.
I20251212 03:12:18.360508 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 125 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:18.360613 20935 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 125 FOLLOWER]: Leader election won for term 125
I20251212 03:12:18.360663 20935 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 125 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:18.360718 20935 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 123.108, Last appended by leader: 108, Current term: 125, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:18.361172 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 124 to 125, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 125 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:18.417444 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 125 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 124. Current term is 125. Ops: []
I20251212 03:12:18.417708 20936 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I20251212 03:12:18.417850 20938 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 124 LEADER]: Stepping down as leader of term 124
I20251212 03:12:18.417894 20938 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 124 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:18.417945 20938 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 124.109, Last appended by leader: 109, Current term: 124, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:18.418025 20938 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 124 FOLLOWER]: Advancing to term 125
I20251212 03:12:18.446704 20758 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 125 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 124 index: 109. Preceding OpId from leader: term: 125 index: 109. (term mismatch)
I20251212 03:12:18.446785 20758 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting all ops after (but not including) 108
I20251212 03:12:18.446800 20758 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting uncommitted NO_OP operation due to leader change: 124.109
I20251212 03:12:18.446815 20758 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 125 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:18.447067 20935 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I20251212 03:12:18.448104 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 125 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 123 index: 108. Preceding OpId from leader: term: 125 index: 109. (index mismatch)
I20251212 03:12:18.448529 20942 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s
I20251212 03:12:18.504942 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:18.549471 20939 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 125 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:18.549579 20939 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 125 FOLLOWER]: Advancing to term 126
I20251212 03:12:18.550258 20939 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 126 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:18.550431 20939 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 126 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:18.550647 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 126 candidate_status { last_received { term: 125 index: 109 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:18.550685 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 126 candidate_status { last_received { term: 125 index: 109 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:18.550875 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 126 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:18.550959 20939 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 126 FOLLOWER]: Leader election lost for term 126. Reason: could not achieve majority
I20251212 03:12:18.564102 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 126 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 125. Current term is 126. Ops: []
I20251212 03:12:18.564383 20944 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 125.109, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20251212 03:12:18.564494 20944 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 125 LEADER]: Stepping down as leader of term 125
I20251212 03:12:18.564514 20944 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 125 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:18.564572 20944 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 125.109, Last appended by leader: 109, Current term: 125, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:18.564662 20944 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 125 FOLLOWER]: Advancing to term 126
I20251212 03:12:18.651687 20938 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 125 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:18.651788 20938 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 125 FOLLOWER]: Advancing to term 126
I20251212 03:12:18.652487 20938 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 126 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:18.652653 20938 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 126 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:18.652866 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 126 candidate_status { last_received { term: 125 index: 109 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:18.652910 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 126 candidate_status { last_received { term: 125 index: 109 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:18.652954 20682 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 126 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 126: Already voted for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in this term.
I20251212 03:12:18.653527 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 126 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 126.
I20251212 03:12:18.653679 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 126 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:18.653777 20938 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 126 FOLLOWER]: Leader election won for term 126
I20251212 03:12:18.653829 20938 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 126 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:18.653877 20938 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 125.109, Last appended by leader: 109, Current term: 126, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:18.654309 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 125 to 126, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 126 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:18.687556 20939 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 126 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:18.687659 20939 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 126 FOLLOWER]: Advancing to term 127
I20251212 03:12:18.688301 20939 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 127 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:18.688476 20939 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 127 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:18.688715 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 127 candidate_status { last_received { term: 125 index: 109 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:18.688727 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 127 candidate_status { last_received { term: 125 index: 109 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:18.688787 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 126 FOLLOWER]: Advancing to term 127
I20251212 03:12:18.689375 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 127 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 127.
I20251212 03:12:18.689604 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 127 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:18.689692 20939 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 127 FOLLOWER]: Leader election won for term 127
I20251212 03:12:18.689746 20939 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 127 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:18.689805 20939 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 125.109, Last appended by leader: 109, Current term: 127, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:18.690356 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 126 to 127, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 127 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:18.735639 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 127 LEADER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 126. Current term is 127. Ops: []
I20251212 03:12:18.735929 20938 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20251212 03:12:18.736040 20938 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer d3fe965dfda94fd4bf3f61a5c0452d63 log is divergent from this leader: its last log entry 127.110 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 109
I20251212 03:12:18.736078 20936 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 126 LEADER]: Stepping down as leader of term 126
I20251212 03:12:18.736120 20936 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 126 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:18.736176 20936 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 126.110, Last appended by leader: 110, Current term: 126, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:18.736251 20936 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 126 FOLLOWER]: Advancing to term 127
I20251212 03:12:18.774852 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 127 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 125 index: 109. Preceding OpId from leader: term: 127 index: 110. (index mismatch)
I20251212 03:12:18.775130 20939 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20251212 03:12:18.776111 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 127 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 126 index: 110. Preceding OpId from leader: term: 127 index: 110. (term mismatch)
I20251212 03:12:18.776167 20759 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting all ops after (but not including) 109
I20251212 03:12:18.776186 20759 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting uncommitted NO_OP operation due to leader change: 126.110
I20251212 03:12:18.776204 20759 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 127 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:18.776417 20939 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20251212 03:12:19.048884 20936 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 127 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:19.048988 20936 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 127 FOLLOWER]: Advancing to term 128
I20251212 03:12:19.049685 20936 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 128 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:19.049853 20936 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 128 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:19.050099 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 128 candidate_status { last_received { term: 127 index: 110 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:19.050107 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 128 candidate_status { last_received { term: 127 index: 110 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:19.050374 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 128 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:19.050472 20936 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 128 FOLLOWER]: Leader election lost for term 128. Reason: could not achieve majority
I20251212 03:12:19.050874 20944 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 127 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:19.050909 20944 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 127 FOLLOWER]: Advancing to term 128
I20251212 03:12:19.051452 20944 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 128 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:19.051586 20944 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 128 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:19.051759 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 128 candidate_status { last_received { term: 127 index: 110 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:19.051761 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 128 candidate_status { last_received { term: 127 index: 110 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:19.051821 20759 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 128 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 in current term 128: Already voted for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in this term.
I20251212 03:12:19.051964 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 128 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:19.052054 20944 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 128 FOLLOWER]: Leader election lost for term 128. Reason: could not achieve majority
I20251212 03:12:19.055699 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 128 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 127. Current term is 128. Ops: []
I20251212 03:12:19.055915 20939 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 127.110, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20251212 03:12:19.056008 20939 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 127 LEADER]: Stepping down as leader of term 127
I20251212 03:12:19.056046 20939 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 127 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:19.056097 20939 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 127.110, Last appended by leader: 110, Current term: 127, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:19.056174 20939 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 127 FOLLOWER]: Advancing to term 128
I20251212 03:12:19.190274 20936 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 128 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:19.190392 20936 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 128 FOLLOWER]: Advancing to term 129
I20251212 03:12:19.191139 20936 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 129 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:19.191309 20936 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 129 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:19.191370 20944 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 128 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:19.191412 20944 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 128 FOLLOWER]: Advancing to term 129
I20251212 03:12:19.191534 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 129 candidate_status { last_received { term: 127 index: 110 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:19.191620 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 128 FOLLOWER]: Advancing to term 129
I20251212 03:12:19.192018 20944 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 129 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:19.192124 20944 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 129 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:19.192178 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 129 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 129.
I20251212 03:12:19.192286 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 129 candidate_status { last_received { term: 127 index: 110 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:19.192291 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 129 candidate_status { last_received { term: 127 index: 110 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:19.192396 20759 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 129 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 in current term 129: Already voted for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in this term.
I20251212 03:12:19.192371 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 129 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 129: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:19.192365 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 129 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters:
I20251212 03:12:19.192394 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 129 candidate_status { last_received { term: 127 index: 110 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:19.192559 20936 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 129 FOLLOWER]: Leader election won for term 129
I20251212 03:12:19.192627 20936 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 129 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:19.192564 20682 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 129 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 in current term 129: Already voted for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in this term.
I20251212 03:12:19.192718 20936 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 127.110, Last appended by leader: 110, Current term: 129, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:19.192797 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 129 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:19.192960 20944 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 129 FOLLOWER]: Leader election lost for term 129. Reason: could not achieve majority
I20251212 03:12:19.193100 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 127 to 129, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 129 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:19.269609 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 129 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 127 index: 110. Preceding OpId from leader: term: 129 index: 111. (index mismatch)
I20251212 03:12:19.269922 20936 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20251212 03:12:19.271052 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 129 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 127 index: 110. Preceding OpId from leader: term: 129 index: 111. (index mismatch)
I20251212 03:12:19.271270 20936 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20251212 03:12:19.371832 20939 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 129 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:19.371922 20939 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 129 FOLLOWER]: Advancing to term 130
I20251212 03:12:19.372611 20939 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 130 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:19.372769 20939 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 130 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:19.372965 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 130 candidate_status { last_received { term: 129 index: 111 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:19.372985 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 130 candidate_status { last_received { term: 129 index: 111 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:19.373209 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 130 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:19.373301 20939 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 130 FOLLOWER]: Leader election lost for term 130. Reason: could not achieve majority
I20251212 03:12:19.394737 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 130 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 129. Current term is 130. Ops: []
I20251212 03:12:19.394973 20949 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 129.111, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20251212 03:12:19.395087 20938 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 129 LEADER]: Stepping down as leader of term 129
I20251212 03:12:19.395130 20938 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 129 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:19.395184 20938 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 129.111, Last appended by leader: 111, Current term: 129, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:19.395294 20938 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 129 FOLLOWER]: Advancing to term 130
I20251212 03:12:19.466801 20944 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 129 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:19.466904 20944 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 129 FOLLOWER]: Advancing to term 130
I20251212 03:12:19.467664 20944 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 130 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:19.467841 20944 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 130 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:19.468094 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 130 candidate_status { last_received { term: 129 index: 111 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:19.468094 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 130 candidate_status { last_received { term: 129 index: 111 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:19.468235 20682 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 130 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 in current term 130: Already voted for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in this term.
I20251212 03:12:19.468761 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 130 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 130.
I20251212 03:12:19.468930 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 130 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:19.469018 20944 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 130 FOLLOWER]: Leader election won for term 130
I20251212 03:12:19.469136 20944 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 130 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:19.469180 20944 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 129.111, Last appended by leader: 111, Current term: 130, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:19.469580 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 129 to 130, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 130 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:19.475535 20939 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 130 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:19.475591 20939 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 130 FOLLOWER]: Advancing to term 131
I20251212 03:12:19.476097 20939 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 131 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:19.476210 20939 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 131 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:19.476400 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 131 candidate_status { last_received { term: 129 index: 111 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:19.476428 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 131 candidate_status { last_received { term: 129 index: 111 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:19.476524 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 130 FOLLOWER]: Advancing to term 131
I20251212 03:12:19.477054 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 131 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 131.
I20251212 03:12:19.477309 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 131 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:19.477406 20939 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 131 FOLLOWER]: Leader election won for term 131
I20251212 03:12:19.477459 20939 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 131 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:19.477511 20939 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 129.111, Last appended by leader: 111, Current term: 131, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:19.477959 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 130 to 131, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 131 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:19.505223 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:19.567407 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 131 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 129 index: 111. Preceding OpId from leader: term: 131 index: 112. (index mismatch)
I20251212 03:12:19.567706 20939 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20251212 03:12:19.568730 20605 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 130 LEADER]: Stepping down as leader of term 130
I20251212 03:12:19.568785 20605 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 130 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:19.568837 20605 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 130.112, Last appended by leader: 112, Current term: 130, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:19.568922 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 130 FOLLOWER]: Advancing to term 131
I20251212 03:12:19.569532 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 131 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 130 index: 112. Preceding OpId from leader: term: 131 index: 112. (term mismatch)
I20251212 03:12:19.569558 20605 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting all ops after (but not including) 111
I20251212 03:12:19.569571 20605 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting uncommitted NO_OP operation due to leader change: 130.112
I20251212 03:12:19.569583 20605 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 131 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:19.569780 20939 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20251212 03:12:19.794355 20944 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 131 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:19.794492 20944 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 131 FOLLOWER]: Advancing to term 132
I20251212 03:12:19.795437 20944 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 132 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:19.795655 20944 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 132 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:19.795827 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 132 candidate_status { last_received { term: 131 index: 112 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:19.795850 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 132 candidate_status { last_received { term: 131 index: 112 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:19.796135 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 132 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:19.796253 20944 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 132 FOLLOWER]: Leader election lost for term 132. Reason: could not achieve majority
I20251212 03:12:19.799479 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 132 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 131. Current term is 132. Ops: []
I20251212 03:12:19.799746 20939 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 131.112, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20251212 03:12:19.799875 20943 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 131 LEADER]: Stepping down as leader of term 131
I20251212 03:12:19.799950 20943 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 131 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:19.800024 20943 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 131.112, Last appended by leader: 112, Current term: 131, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:19.800124 20943 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 131 FOLLOWER]: Advancing to term 132
I20251212 03:12:19.837399 20938 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 131 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:19.837489 20938 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 131 FOLLOWER]: Advancing to term 132
I20251212 03:12:19.838148 20938 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 132 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:19.838315 20938 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 132 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:19.838519 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 132 candidate_status { last_received { term: 131 index: 112 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:19.838682 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 132 candidate_status { last_received { term: 131 index: 112 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:19.838747 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 132 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 132: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:19.839185 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 132 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 132.
I20251212 03:12:19.839359 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 132 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:19.839440 20938 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 132 FOLLOWER]: Leader election won for term 132
I20251212 03:12:19.839494 20938 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 132 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:19.839541 20938 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 131.112, Last appended by leader: 112, Current term: 132, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:19.840008 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 131 to 132, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 132 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:19.922528 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 132 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 131 index: 112. Preceding OpId from leader: term: 132 index: 113. (index mismatch)
I20251212 03:12:19.922781 20938 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20251212 03:12:19.923760 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 132 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 131 index: 112. Preceding OpId from leader: term: 132 index: 113. (index mismatch)
I20251212 03:12:19.923947 20951 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20251212 03:12:20.025472 20944 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 132 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:20.025585 20944 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 132 FOLLOWER]: Advancing to term 133
I20251212 03:12:20.026314 20943 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 132 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:20.026396 20943 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 132 FOLLOWER]: Advancing to term 133
I20251212 03:12:20.026386 20944 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 133 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:20.026569 20944 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 133 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:20.026772 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 133 candidate_status { last_received { term: 132 index: 113 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:20.026777 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 133 candidate_status { last_received { term: 132 index: 113 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:20.027248 20943 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 133 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:20.027410 20682 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 133 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 in current term 133: Already voted for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in this term.
I20251212 03:12:20.027431 20943 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 133 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:20.027632 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 133 candidate_status { last_received { term: 132 index: 113 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:20.027667 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 133 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:20.027700 20606 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 133 candidate_status { last_received { term: 132 index: 113 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:20.027757 20606 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 133 FOLLOWER]: Leader election vote request: Denying vote to candidate d3fe965dfda94fd4bf3f61a5c0452d63 in current term 133: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:20.027796 20944 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 133 FOLLOWER]: Leader election lost for term 133. Reason: could not achieve majority
I20251212 03:12:20.027844 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 133 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 132. Current term is 133. Ops: []
I20251212 03:12:20.027937 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 133 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:20.028028 20943 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 133 FOLLOWER]: Leader election lost for term 133. Reason: could not achieve majority
I20251212 03:12:20.028034 20949 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 132.113, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20251212 03:12:20.028172 20949 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 132 LEADER]: Stepping down as leader of term 132
I20251212 03:12:20.028205 20949 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 132 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:20.028263 20949 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 113, Committed index: 113, Last appended: 132.113, Last appended by leader: 113, Current term: 132, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:20.028349 20949 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 132 FOLLOWER]: Advancing to term 133
I20251212 03:12:20.131295 20944 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 133 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:20.131397 20944 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 133 FOLLOWER]: Advancing to term 134
I20251212 03:12:20.132072 20944 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 134 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:20.132357 20944 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 134 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:20.132510 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 134 candidate_status { last_received { term: 132 index: 113 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:20.132601 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 133 FOLLOWER]: Advancing to term 134
I20251212 03:12:20.132517 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 134 candidate_status { last_received { term: 132 index: 113 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:20.132699 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 133 FOLLOWER]: Advancing to term 134
I20251212 03:12:20.133186 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 134 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 134.
I20251212 03:12:20.133214 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 134 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 134.
I20251212 03:12:20.133420 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 134 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters:
I20251212 03:12:20.133553 20944 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 134 FOLLOWER]: Leader election won for term 134
I20251212 03:12:20.133680 20944 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 134 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:20.133737 20944 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 113, Committed index: 113, Last appended: 132.113, Last appended by leader: 113, Current term: 134, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:20.134323 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 132 to 134, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 134 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:20.228782 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 134 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 132 index: 113. Preceding OpId from leader: term: 134 index: 114. (index mismatch)
I20251212 03:12:20.229064 20944 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20251212 03:12:20.230144 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 134 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 132 index: 113. Preceding OpId from leader: term: 134 index: 114. (index mismatch)
I20251212 03:12:20.230449 20952 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20251212 03:12:20.407117 20943 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 134 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:20.407219 20943 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 134 FOLLOWER]: Advancing to term 135
I20251212 03:12:20.408092 20943 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 135 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:20.408475 20943 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 135 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:20.408710 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 135 candidate_status { last_received { term: 134 index: 114 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:20.408721 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 135 candidate_status { last_received { term: 134 index: 114 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:20.408949 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 135 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:20.409008 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 135 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 134. Current term is 135. Ops: []
I20251212 03:12:20.409056 20943 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 135 FOLLOWER]: Leader election lost for term 135. Reason: could not achieve majority
I20251212 03:12:20.409191 20953 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 134.114, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20251212 03:12:20.409296 20953 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 134 LEADER]: Stepping down as leader of term 134
I20251212 03:12:20.409332 20953 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 134 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:20.409399 20953 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 134.114, Last appended by leader: 114, Current term: 134, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:20.409478 20953 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 134 FOLLOWER]: Advancing to term 135
I20251212 03:12:20.482178 20949 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 134 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:20.482334 20949 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 134 FOLLOWER]: Advancing to term 135
I20251212 03:12:20.483026 20949 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 135 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:20.483192 20949 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 135 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:20.483465 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 135 candidate_status { last_received { term: 134 index: 114 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:20.483551 20682 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 135 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 135: Already voted for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in this term.
I20251212 03:12:20.483476 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 135 candidate_status { last_received { term: 134 index: 114 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:20.484155 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 135 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 135.
I20251212 03:12:20.484418 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 135 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:20.484561 20949 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 135 FOLLOWER]: Leader election won for term 135
I20251212 03:12:20.484627 20949 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 135 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:20.484694 20949 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 134.114, Last appended by leader: 114, Current term: 135, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:20.485200 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 134 to 135, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 135 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:20.505478 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:20.517508 20943 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 135 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:20.517628 20943 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 135 FOLLOWER]: Advancing to term 136
I20251212 03:12:20.518574 20943 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 136 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:20.518756 20943 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 136 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:20.519004 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 136 candidate_status { last_received { term: 134 index: 114 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:20.519078 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 135 FOLLOWER]: Advancing to term 136
I20251212 03:12:20.519011 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 136 candidate_status { last_received { term: 134 index: 114 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:20.519599 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 136 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 136.
I20251212 03:12:20.519821 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 136 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:20.519923 20943 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 136 FOLLOWER]: Leader election won for term 136
I20251212 03:12:20.520027 20943 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 136 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:20.520087 20943 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 134.114, Last appended by leader: 114, Current term: 136, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:20.520598 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 135 to 136, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 136 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:20.576742 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 136 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 135. Current term is 136. Ops: []
I20251212 03:12:20.577052 20949 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20251212 03:12:20.577162 20949 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 135 LEADER]: Stepping down as leader of term 135
I20251212 03:12:20.577207 20949 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 135 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:20.577320 20949 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 135.115, Last appended by leader: 115, Current term: 135, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:20.577399 20949 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 135 FOLLOWER]: Advancing to term 136
I20251212 03:12:20.622044 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 136 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 134 index: 114. Preceding OpId from leader: term: 136 index: 115. (index mismatch)
I20251212 03:12:20.622406 20943 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20251212 03:12:20.623529 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 136 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 135 index: 115. Preceding OpId from leader: term: 136 index: 115. (term mismatch)
I20251212 03:12:20.623587 20759 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting all ops after (but not including) 114
I20251212 03:12:20.623611 20759 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting uncommitted NO_OP operation due to leader change: 135.115
I20251212 03:12:20.623632 20759 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 136 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:20.623827 20955 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20251212 03:12:20.822682 20949 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 136 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:20.822793 20949 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 136 FOLLOWER]: Advancing to term 137
I20251212 03:12:20.823518 20949 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 137 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:20.823657 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 137 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 136. Current term is 137. Ops: []
I20251212 03:12:20.823798 20949 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 137 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:20.823901 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 137 candidate_status { last_received { term: 136 index: 115 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:20.823994 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 137 candidate_status { last_received { term: 136 index: 115 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:20.824091 20943 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 136.115, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20251212 03:12:20.824182 20943 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 136 LEADER]: Stepping down as leader of term 136
I20251212 03:12:20.824215 20943 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 136 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:20.824290 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 137 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:20.824280 20943 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 136.115, Last appended by leader: 115, Current term: 136, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:20.824412 20949 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 137 FOLLOWER]: Leader election lost for term 137. Reason: could not achieve majority
I20251212 03:12:20.824416 20943 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 136 FOLLOWER]: Advancing to term 137
I20251212 03:12:20.887362 20953 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 136 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:20.887460 20953 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 136 FOLLOWER]: Advancing to term 137
I20251212 03:12:20.888160 20953 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 137 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:20.888427 20953 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 137 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:20.888545 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 137 candidate_status { last_received { term: 136 index: 115 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:20.888545 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 137 candidate_status { last_received { term: 136 index: 115 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:20.888682 20759 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 137 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 in current term 137: Already voted for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in this term.
I20251212 03:12:20.889221 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 137 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 137.
I20251212 03:12:20.889407 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 137 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:20.889500 20953 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 137 FOLLOWER]: Leader election won for term 137
I20251212 03:12:20.889555 20953 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 137 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:20.889616 20953 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 136.115, Last appended by leader: 115, Current term: 137, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:20.890062 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 136 to 137, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 137 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:20.970827 20949 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 137 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:20.970933 20949 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 137 FOLLOWER]: Advancing to term 138
I20251212 03:12:20.971669 20949 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 138 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:20.971829 20949 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 138 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:20.972046 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 138 candidate_status { last_received { term: 136 index: 115 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:20.972046 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 138 candidate_status { last_received { term: 136 index: 115 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:20.972175 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 137 FOLLOWER]: Advancing to term 138
I20251212 03:12:20.972800 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 138 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 138.
I20251212 03:12:20.972996 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 138 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:20.973085 20949 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 138 FOLLOWER]: Leader election won for term 138
I20251212 03:12:20.973138 20949 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 138 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:20.973196 20949 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 136.115, Last appended by leader: 115, Current term: 138, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:20.973763 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 137 to 138, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 138 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:20.980582 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 138 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 137. Current term is 138. Ops: []
I20251212 03:12:20.980887 20953 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20251212 03:12:20.981010 20953 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 137 LEADER]: Stepping down as leader of term 137
I20251212 03:12:20.981055 20953 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 137 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:20.981112 20953 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 137.116, Last appended by leader: 116, Current term: 137, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:20.981190 20953 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 137 FOLLOWER]: Advancing to term 138
I20251212 03:12:21.064152 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 138 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 136 index: 115. Preceding OpId from leader: term: 138 index: 116. (index mismatch)
I20251212 03:12:21.064493 20949 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20251212 03:12:21.065591 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 138 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 137 index: 116. Preceding OpId from leader: term: 138 index: 116. (term mismatch)
I20251212 03:12:21.065646 20605 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting all ops after (but not including) 115
I20251212 03:12:21.065665 20605 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting uncommitted NO_OP operation due to leader change: 137.116
I20251212 03:12:21.065687 20605 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 138 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:21.065900 20956 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20251212 03:12:21.166409 20943 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 138 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:21.166527 20943 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 138 FOLLOWER]: Advancing to term 139
I20251212 03:12:21.167236 20943 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 139 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:21.167552 20943 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 139 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:21.167635 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 139 candidate_status { last_received { term: 138 index: 116 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:21.167742 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 139 candidate_status { last_received { term: 138 index: 116 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:21.167848 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 138 FOLLOWER]: Advancing to term 139
I20251212 03:12:21.168435 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 139 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 139.
I20251212 03:12:21.168627 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 139 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:21.168740 20943 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 139 FOLLOWER]: Leader election won for term 139
I20251212 03:12:21.168869 20943 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 139 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:21.168943 20943 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 138.116, Last appended by leader: 116, Current term: 139, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:21.169409 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 138 to 139, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 139 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:21.172340 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 139 LEADER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 138. Current term is 139. Ops: []
I20251212 03:12:21.172571 20956 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 138.116, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20251212 03:12:21.172662 20956 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer d3fe965dfda94fd4bf3f61a5c0452d63 log is divergent from this leader: its last log entry 139.117 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 116
I20251212 03:12:21.172703 20949 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 138 LEADER]: Stepping down as leader of term 138
I20251212 03:12:21.172752 20949 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 138 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:21.172808 20949 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 138.116, Last appended by leader: 116, Current term: 138, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:21.172885 20949 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 138 FOLLOWER]: Advancing to term 139
I20251212 03:12:21.246872 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 139 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 138 index: 116. Preceding OpId from leader: term: 139 index: 117. (index mismatch)
I20251212 03:12:21.247220 20943 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20251212 03:12:21.248561 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 139 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 138 index: 116. Preceding OpId from leader: term: 139 index: 117. (index mismatch)
I20251212 03:12:21.248842 20955 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20251212 03:12:21.505733 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:21.523722 20944 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 139 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:21.523828 20944 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 139 FOLLOWER]: Advancing to term 140
I20251212 03:12:21.524564 20944 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 140 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:21.524731 20944 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 140 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:21.524979 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 140 candidate_status { last_received { term: 139 index: 117 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:21.524994 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 140 candidate_status { last_received { term: 139 index: 117 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:21.525271 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 140 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:21.525414 20944 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 140 FOLLOWER]: Leader election lost for term 140. Reason: could not achieve majority
I20251212 03:12:21.535365 20956 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 139 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:21.535447 20956 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 139 FOLLOWER]: Advancing to term 140
I20251212 03:12:21.536159 20956 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 140 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:21.536407 20956 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 140 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:21.536504 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 140 candidate_status { last_received { term: 139 index: 117 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:21.536556 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 140 candidate_status { last_received { term: 139 index: 117 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:21.536619 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 140 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 140: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:21.536787 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 140 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:21.536881 20956 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 140 FOLLOWER]: Leader election lost for term 140. Reason: could not achieve majority
I20251212 03:12:21.537715 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 140 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 139. Current term is 140. Ops: []
I20251212 03:12:21.537959 20957 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 139.117, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20251212 03:12:21.538064 20957 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 139 LEADER]: Stepping down as leader of term 139
I20251212 03:12:21.538100 20957 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 139 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:21.538166 20957 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 139.117, Last appended by leader: 117, Current term: 139, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:21.538277 20957 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 139 FOLLOWER]: Advancing to term 140
I20251212 03:12:21.650355 20956 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 140 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:21.650470 20956 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 140 FOLLOWER]: Advancing to term 141
I20251212 03:12:21.651190 20956 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 141 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:21.651377 20956 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 141 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:21.651602 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 141 candidate_status { last_received { term: 139 index: 117 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:21.651614 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 141 candidate_status { last_received { term: 139 index: 117 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:21.651671 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 140 FOLLOWER]: Advancing to term 141
I20251212 03:12:21.651721 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 140 FOLLOWER]: Advancing to term 141
I20251212 03:12:21.652235 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 141 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 141.
I20251212 03:12:21.652349 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 141 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 141.
I20251212 03:12:21.652441 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 141 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters:
I20251212 03:12:21.652563 20956 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 141 FOLLOWER]: Leader election won for term 141
I20251212 03:12:21.652633 20956 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 141 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:21.652691 20956 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 139.117, Last appended by leader: 117, Current term: 141, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:21.653154 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 139 to 141, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 141 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:21.729796 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 141 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 139 index: 117. Preceding OpId from leader: term: 141 index: 118. (index mismatch)
I20251212 03:12:21.730064 20956 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20251212 03:12:21.731453 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 141 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 139 index: 117. Preceding OpId from leader: term: 141 index: 118. (index mismatch)
I20251212 03:12:21.731686 20949 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20251212 03:12:21.933269 20944 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 141 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:21.933365 20944 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 141 FOLLOWER]: Advancing to term 142
I20251212 03:12:21.934010 20944 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 142 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:21.934181 20944 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 142 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:21.934388 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 142 candidate_status { last_received { term: 141 index: 118 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:21.934516 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 142 candidate_status { last_received { term: 141 index: 118 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:21.934644 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 142 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:21.934767 20944 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 142 FOLLOWER]: Leader election lost for term 142. Reason: could not achieve majority
I20251212 03:12:21.939105 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 142 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 141. Current term is 142. Ops: []
I20251212 03:12:21.939360 20956 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 141.118, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20251212 03:12:21.939474 20956 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 141 LEADER]: Stepping down as leader of term 141
I20251212 03:12:21.939509 20956 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 141 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:21.939565 20956 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 141.118, Last appended by leader: 118, Current term: 141, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:21.939635 20956 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 141 FOLLOWER]: Advancing to term 142
I20251212 03:12:22.005882 20957 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 141 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:22.005990 20957 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 141 FOLLOWER]: Advancing to term 142
I20251212 03:12:22.006783 20957 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 142 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:22.006944 20957 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 142 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:22.007160 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 142 candidate_status { last_received { term: 141 index: 118 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:22.007160 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 142 candidate_status { last_received { term: 141 index: 118 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:22.007277 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 142 FOLLOWER]: Leader election vote request: Denying vote to candidate d3fe965dfda94fd4bf3f61a5c0452d63 in current term 142: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:22.007834 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 142 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 142.
I20251212 03:12:22.008018 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 142 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:22.008138 20957 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 142 FOLLOWER]: Leader election won for term 142
I20251212 03:12:22.008195 20957 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 142 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:22.008260 20957 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 141.118, Last appended by leader: 118, Current term: 142, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:22.008888 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 141 to 142, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 142 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:22.077772 20944 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 142 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:22.077899 20944 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 142 FOLLOWER]: Advancing to term 143
I20251212 03:12:22.078629 20944 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 143 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:22.078810 20944 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 143 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:22.079047 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 143 candidate_status { last_received { term: 141 index: 118 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:22.079047 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 143 candidate_status { last_received { term: 141 index: 118 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:22.079178 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 142 FOLLOWER]: Advancing to term 143
I20251212 03:12:22.079780 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 143 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 143.
I20251212 03:12:22.079950 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 143 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:22.080080 20944 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 143 FOLLOWER]: Leader election won for term 143
I20251212 03:12:22.080291 20944 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 143 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:22.080363 20944 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 141.118, Last appended by leader: 118, Current term: 143, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:22.080811 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 142 to 143, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 143 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:22.085157 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 143 LEADER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 142. Current term is 143. Ops: []
I20251212 03:12:22.085414 20957 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20251212 03:12:22.085496 20957 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer e9d27948dced4eba8f8228ddc4529393 log is divergent from this leader: its last log entry 143.119 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 118
I20251212 03:12:22.085533 20943 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 142 LEADER]: Stepping down as leader of term 142
I20251212 03:12:22.085575 20943 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 142 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:22.085624 20943 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 142.119, Last appended by leader: 119, Current term: 142, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:22.085698 20943 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 142 FOLLOWER]: Advancing to term 143
I20251212 03:12:22.165179 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 143 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 142 index: 119. Preceding OpId from leader: term: 143 index: 119. (term mismatch)
I20251212 03:12:22.165258 20682 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting all ops after (but not including) 118
I20251212 03:12:22.165272 20682 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting uncommitted NO_OP operation due to leader change: 142.119
I20251212 03:12:22.165293 20682 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 143 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:22.165509 20944 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20251212 03:12:22.166584 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 143 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 141 index: 118. Preceding OpId from leader: term: 143 index: 119. (index mismatch)
I20251212 03:12:22.166806 20959 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20251212 03:12:22.267570 20956 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 143 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:22.267678 20956 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 143 FOLLOWER]: Advancing to term 144
I20251212 03:12:22.268368 20956 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 144 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:22.268548 20956 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 144 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:22.268802 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 144 candidate_status { last_received { term: 143 index: 119 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:22.268802 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 144 candidate_status { last_received { term: 143 index: 119 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:22.268932 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 143 FOLLOWER]: Advancing to term 144
I20251212 03:12:22.269580 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 144 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 144.
I20251212 03:12:22.269785 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 144 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:22.269907 20956 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 144 FOLLOWER]: Leader election won for term 144
I20251212 03:12:22.269980 20956 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 144 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:22.270047 20956 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 143.119, Last appended by leader: 119, Current term: 144, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:22.270556 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 143 to 144, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 144 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:22.271564 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 144 LEADER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 143. Current term is 144. Ops: []
I20251212 03:12:22.271728 20958 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 143.119, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20251212 03:12:22.271814 20958 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer f894b0d567ab4a2ebdcb416d04ef48e5 log is divergent from this leader: its last log entry 144.120 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 119
I20251212 03:12:22.271847 20944 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 143 LEADER]: Stepping down as leader of term 143
I20251212 03:12:22.271883 20944 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 143 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:22.271935 20944 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 143.119, Last appended by leader: 119, Current term: 143, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:22.271987 20944 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 143 FOLLOWER]: Advancing to term 144
I20251212 03:12:22.361594 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 144 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 143 index: 119. Preceding OpId from leader: term: 144 index: 120. (index mismatch)
I20251212 03:12:22.362030 20956 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20251212 03:12:22.363512 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 144 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 143 index: 119. Preceding OpId from leader: term: 144 index: 120. (index mismatch)
I20251212 03:12:22.363864 20960 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20251212 03:12:22.465557 20957 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 144 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:22.465662 20957 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 144 FOLLOWER]: Advancing to term 145
I20251212 03:12:22.466387 20957 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 145 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:22.466581 20957 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 145 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:22.466811 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 145 candidate_status { last_received { term: 144 index: 120 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:22.466835 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 145 candidate_status { last_received { term: 144 index: 120 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:22.467043 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 145 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:22.467139 20957 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 145 FOLLOWER]: Leader election lost for term 145. Reason: could not achieve majority
I20251212 03:12:22.469524 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 145 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 144. Current term is 145. Ops: []
I20251212 03:12:22.469739 20960 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 144.120, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20251212 03:12:22.469858 20960 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 144 LEADER]: Stepping down as leader of term 144
I20251212 03:12:22.469887 20960 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 144 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:22.469945 20960 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 144.120, Last appended by leader: 120, Current term: 144, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:22.470042 20960 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 144 FOLLOWER]: Advancing to term 145
I20251212 03:12:22.506009 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:22.546727 20944 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 144 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:22.546836 20944 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 144 FOLLOWER]: Advancing to term 145
I20251212 03:12:22.547537 20944 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 145 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:22.547699 20944 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 145 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:22.547933 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 145 candidate_status { last_received { term: 144 index: 120 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:22.547933 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 145 candidate_status { last_received { term: 144 index: 120 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:22.548051 20682 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 145 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 in current term 145: Already voted for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in this term.
I20251212 03:12:22.548605 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 145 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 145.
I20251212 03:12:22.548820 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 145 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:22.548928 20944 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 145 FOLLOWER]: Leader election won for term 145
I20251212 03:12:22.548982 20944 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 145 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:22.549026 20944 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 144.120, Last appended by leader: 120, Current term: 145, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:22.549515 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 144 to 145, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 145 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:22.568528 20957 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 145 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:22.568629 20957 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 145 FOLLOWER]: Advancing to term 146
I20251212 03:12:22.569271 20957 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 146 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:22.569430 20957 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 146 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:22.569635 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 146 candidate_status { last_received { term: 144 index: 120 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:22.569644 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 146 candidate_status { last_received { term: 144 index: 120 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:22.569705 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 145 FOLLOWER]: Advancing to term 146
I20251212 03:12:22.570329 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 146 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 146.
I20251212 03:12:22.570540 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 146 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:22.570642 20957 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 146 FOLLOWER]: Leader election won for term 146
I20251212 03:12:22.570719 20957 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 146 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:22.570781 20957 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 144.120, Last appended by leader: 120, Current term: 146, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:22.571224 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 145 to 146, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 146 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:22.649042 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 146 LEADER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 145. Current term is 146. Ops: []
I20251212 03:12:22.649417 20944 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20251212 03:12:22.649545 20944 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer d3fe965dfda94fd4bf3f61a5c0452d63 log is divergent from this leader: its last log entry 146.121 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 120
I20251212 03:12:22.649582 20958 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 145 LEADER]: Stepping down as leader of term 145
I20251212 03:12:22.649609 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 146 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 144 index: 120. Preceding OpId from leader: term: 146 index: 121. (index mismatch)
I20251212 03:12:22.649626 20958 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 145 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:22.649734 20958 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 145.121, Last appended by leader: 121, Current term: 145, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:22.649822 20958 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 145 FOLLOWER]: Advancing to term 146
I20251212 03:12:22.649832 20957 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20251212 03:12:22.650985 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 146 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 145 index: 121. Preceding OpId from leader: term: 146 index: 121. (term mismatch)
I20251212 03:12:22.651036 20605 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting all ops after (but not including) 120
I20251212 03:12:22.651068 20605 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting uncommitted NO_OP operation due to leader change: 145.121
I20251212 03:12:22.651090 20605 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 146 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:22.651264 20943 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20251212 03:12:22.751328 20960 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 146 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:22.751442 20960 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 146 FOLLOWER]: Advancing to term 147
I20251212 03:12:22.752153 20960 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 147 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:22.752362 20960 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 147 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:22.752619 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 147 candidate_status { last_received { term: 146 index: 121 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:22.752626 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 147 candidate_status { last_received { term: 146 index: 121 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:22.752709 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 146 FOLLOWER]: Advancing to term 147
I20251212 03:12:22.753400 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 147 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 147.
I20251212 03:12:22.753580 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 147 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:22.753688 20960 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 147 FOLLOWER]: Leader election won for term 147
I20251212 03:12:22.753751 20960 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 147 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:22.753811 20960 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 146.121, Last appended by leader: 121, Current term: 147, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:22.754293 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 146 to 147, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 147 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:22.755887 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 147 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 146. Current term is 147. Ops: []
I20251212 03:12:22.756104 20961 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 146.121, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20251212 03:12:22.756197 20961 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 146 LEADER]: Stepping down as leader of term 146
I20251212 03:12:22.756232 20961 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 146 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:22.756283 20961 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 146.121, Last appended by leader: 121, Current term: 146, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:22.756342 20961 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 146 FOLLOWER]: Advancing to term 147
I20251212 03:12:22.848886 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 147 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 146 index: 121. Preceding OpId from leader: term: 147 index: 122. (index mismatch)
I20251212 03:12:22.849189 20960 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20251212 03:12:22.850178 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 147 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 146 index: 121. Preceding OpId from leader: term: 147 index: 122. (index mismatch)
I20251212 03:12:22.850416 20960 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20251212 03:12:22.951609 20958 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 147 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:22.951712 20958 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 147 FOLLOWER]: Advancing to term 148
I20251212 03:12:22.952415 20958 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 148 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:22.952582 20958 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 148 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:22.952852 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 148 candidate_status { last_received { term: 147 index: 122 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:22.952852 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 148 candidate_status { last_received { term: 147 index: 122 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:22.953130 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 148 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:22.953207 20958 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 148 FOLLOWER]: Leader election lost for term 148. Reason: could not achieve majority
I20251212 03:12:22.973902 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 148 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 147. Current term is 148. Ops: []
I20251212 03:12:22.974179 20949 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 147.122, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20251212 03:12:22.974332 20949 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 147 LEADER]: Stepping down as leader of term 147
I20251212 03:12:22.974366 20949 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 147 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:22.974426 20949 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 147.122, Last appended by leader: 122, Current term: 147, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:22.974509 20949 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 147 FOLLOWER]: Advancing to term 148
I20251212 03:12:23.046566 20961 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 147 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:23.046674 20961 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 147 FOLLOWER]: Advancing to term 148
I20251212 03:12:23.047371 20961 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 148 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:23.047547 20961 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 148 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:23.047770 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 148 candidate_status { last_received { term: 147 index: 122 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:23.047890 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 148 candidate_status { last_received { term: 147 index: 122 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:23.047958 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 148 FOLLOWER]: Leader election vote request: Denying vote to candidate d3fe965dfda94fd4bf3f61a5c0452d63 in current term 148: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:23.048492 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 148 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 148.
I20251212 03:12:23.048677 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 148 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:23.048764 20961 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 148 FOLLOWER]: Leader election won for term 148
I20251212 03:12:23.048816 20961 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 148 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:23.048878 20961 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 147.122, Last appended by leader: 122, Current term: 148, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:23.049415 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 147 to 148, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 148 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:23.060673 20958 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 148 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:23.060768 20958 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 148 FOLLOWER]: Advancing to term 149
I20251212 03:12:23.061488 20958 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 149 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:23.061633 20958 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 149 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:23.061858 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 149 candidate_status { last_received { term: 147 index: 122 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:23.061858 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 149 candidate_status { last_received { term: 147 index: 122 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:23.061990 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 148 FOLLOWER]: Advancing to term 149
I20251212 03:12:23.062600 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 149 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 149.
I20251212 03:12:23.062812 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 149 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:23.062908 20958 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 149 FOLLOWER]: Leader election won for term 149
I20251212 03:12:23.062958 20958 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 149 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:23.063017 20958 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 147.122, Last appended by leader: 122, Current term: 149, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:23.063472 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 148 to 149, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 149 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:23.152909 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 149 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 148. Current term is 149. Ops: []
I20251212 03:12:23.153338 20961 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20251212 03:12:23.153473 20961 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 148 LEADER]: Stepping down as leader of term 148
I20251212 03:12:23.153508 20961 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 148 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:23.153592 20961 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 148.123, Last appended by leader: 123, Current term: 148, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:23.153672 20961 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 148 FOLLOWER]: Advancing to term 149
I20251212 03:12:23.170960 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 149 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 147 index: 122. Preceding OpId from leader: term: 149 index: 123. (index mismatch)
I20251212 03:12:23.171253 20958 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20251212 03:12:23.172267 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 149 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 148 index: 123. Preceding OpId from leader: term: 149 index: 123. (term mismatch)
I20251212 03:12:23.172322 20682 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting all ops after (but not including) 122
I20251212 03:12:23.172348 20682 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting uncommitted NO_OP operation due to leader change: 148.123
I20251212 03:12:23.172369 20682 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 149 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:23.172585 20958 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20251212 03:12:23.360503 20949 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 149 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:23.360606 20949 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 149 FOLLOWER]: Advancing to term 150
I20251212 03:12:23.361320 20949 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 150 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:23.361497 20949 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 150 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:23.361717 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 150 candidate_status { last_received { term: 149 index: 123 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:23.361721 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 150 candidate_status { last_received { term: 149 index: 123 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:23.361966 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 150 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:23.362058 20949 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 150 FOLLOWER]: Leader election lost for term 150. Reason: could not achieve majority
I20251212 03:12:23.379455 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 150 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 149. Current term is 150. Ops: []
I20251212 03:12:23.379769 20944 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 149.123, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20251212 03:12:23.379899 20944 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 149 LEADER]: Stepping down as leader of term 149
I20251212 03:12:23.379937 20944 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 149 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:23.379999 20944 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 149.123, Last appended by leader: 123, Current term: 149, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:23.380079 20944 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 149 FOLLOWER]: Advancing to term 150
I20251212 03:12:23.447397 20957 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 149 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:23.447494 20957 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 149 FOLLOWER]: Advancing to term 150
I20251212 03:12:23.448165 20957 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 150 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:23.448325 20957 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 150 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:23.448561 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 150 candidate_status { last_received { term: 149 index: 123 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:23.448565 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 150 candidate_status { last_received { term: 149 index: 123 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:23.448662 20759 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 150 FOLLOWER]: Leader election vote request: Denying vote to candidate d3fe965dfda94fd4bf3f61a5c0452d63 in current term 150: Already voted for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in this term.
I20251212 03:12:23.449156 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 150 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 150.
I20251212 03:12:23.449340 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 150 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:23.449450 20957 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 150 FOLLOWER]: Leader election won for term 150
I20251212 03:12:23.449517 20957 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 150 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:23.449559 20957 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 149.123, Last appended by leader: 123, Current term: 150, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:23.450009 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 149 to 150, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 150 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:23.505136 20949 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 150 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:23.505237 20949 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 150 FOLLOWER]: Advancing to term 151
I20251212 03:12:23.505860 20949 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 151 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:23.506008 20949 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 151 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:23.506286 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 151 candidate_status { last_received { term: 149 index: 123 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:23.506251 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:23.506251 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 151 candidate_status { last_received { term: 149 index: 123 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:23.506419 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 150 FOLLOWER]: Advancing to term 151
I20251212 03:12:23.507014 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 151 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 151.
I20251212 03:12:23.507194 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 151 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:23.507275 20949 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 151 FOLLOWER]: Leader election won for term 151
I20251212 03:12:23.507383 20949 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 151 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:23.507447 20949 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 149.123, Last appended by leader: 123, Current term: 151, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:23.507900 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 150 to 151, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 151 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:23.529515 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 151 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 150. Current term is 151. Ops: []
I20251212 03:12:23.529757 20957 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20251212 03:12:23.529867 20957 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 150 LEADER]: Stepping down as leader of term 150
I20251212 03:12:23.529904 20957 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 150 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:23.529971 20957 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 150.124, Last appended by leader: 124, Current term: 150, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:23.530035 20957 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 150 FOLLOWER]: Advancing to term 151
I20251212 03:12:23.596472 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 151 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 149 index: 123. Preceding OpId from leader: term: 151 index: 124. (index mismatch)
I20251212 03:12:23.596801 20949 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20251212 03:12:23.598093 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 151 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 150 index: 124. Preceding OpId from leader: term: 151 index: 124. (term mismatch)
I20251212 03:12:23.598145 20682 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting all ops after (but not including) 123
I20251212 03:12:23.598167 20682 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting uncommitted NO_OP operation due to leader change: 150.124
I20251212 03:12:23.598186 20682 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 151 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:23.598356 20949 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20251212 03:12:23.698719 20944 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 151 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:23.698832 20944 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 151 FOLLOWER]: Advancing to term 152
I20251212 03:12:23.699551 20944 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 152 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:23.699700 20944 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 152 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:23.699857 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 152 candidate_status { last_received { term: 151 index: 124 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:23.699868 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 152 candidate_status { last_received { term: 151 index: 124 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:23.700106 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 152 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:23.700192 20944 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 152 FOLLOWER]: Leader election lost for term 152. Reason: could not achieve majority
I20251212 03:12:23.716796 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 152 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 151. Current term is 152. Ops: []
I20251212 03:12:23.717036 20963 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 151.124, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20251212 03:12:23.717175 20963 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 151 LEADER]: Stepping down as leader of term 151
I20251212 03:12:23.717213 20963 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 151 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:23.717334 20963 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 151.124, Last appended by leader: 124, Current term: 151, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:23.717420 20963 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 151 FOLLOWER]: Advancing to term 152
I20251212 03:12:23.799743 20957 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 151 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:23.799860 20957 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 151 FOLLOWER]: Advancing to term 152
I20251212 03:12:23.800591 20957 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 152 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:23.800779 20957 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 152 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:23.801065 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 152 candidate_status { last_received { term: 151 index: 124 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:23.801087 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 152 candidate_status { last_received { term: 151 index: 124 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:23.801234 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 152 FOLLOWER]: Leader election vote request: Denying vote to candidate d3fe965dfda94fd4bf3f61a5c0452d63 in current term 152: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:23.801823 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 152 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 152.
I20251212 03:12:23.802052 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 152 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:23.802150 20957 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 152 FOLLOWER]: Leader election won for term 152
I20251212 03:12:23.802233 20957 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 152 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:23.802309 20957 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 151.124, Last appended by leader: 124, Current term: 152, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:23.802817 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 151 to 152, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 152 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:23.807585 20944 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 152 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:23.807649 20944 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 152 FOLLOWER]: Advancing to term 153
I20251212 03:12:23.808194 20944 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 153 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:23.808328 20944 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 153 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:23.808512 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 153 candidate_status { last_received { term: 151 index: 124 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:23.808596 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 152 FOLLOWER]: Advancing to term 153
I20251212 03:12:23.808516 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 153 candidate_status { last_received { term: 151 index: 124 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:23.809128 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 153 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 153.
I20251212 03:12:23.809366 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 153 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:23.809486 20944 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 153 FOLLOWER]: Leader election won for term 153
I20251212 03:12:23.809545 20944 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 153 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:23.809587 20944 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 151.124, Last appended by leader: 124, Current term: 153, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:23.810014 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 152 to 153, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 153 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:23.888360 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 153 LEADER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 152. Current term is 153. Ops: []
I20251212 03:12:23.888712 20957 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20251212 03:12:23.888831 20957 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer e9d27948dced4eba8f8228ddc4529393 log is divergent from this leader: its last log entry 153.125 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 124
I20251212 03:12:23.888876 20961 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 152 LEADER]: Stepping down as leader of term 152
I20251212 03:12:23.889014 20961 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 152 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:23.889117 20961 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 152.125, Last appended by leader: 125, Current term: 152, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:23.889259 20961 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 152 FOLLOWER]: Advancing to term 153
I20251212 03:12:23.905535 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 153 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 152 index: 125. Preceding OpId from leader: term: 153 index: 125. (term mismatch)
I20251212 03:12:23.905611 20682 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting all ops after (but not including) 124
I20251212 03:12:23.905627 20682 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: Aborting uncommitted NO_OP operation due to leader change: 152.125
I20251212 03:12:23.905642 20682 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 153 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:23.905858 20944 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20251212 03:12:23.906971 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 153 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 151 index: 124. Preceding OpId from leader: term: 153 index: 125. (index mismatch)
I20251212 03:12:23.907182 20962 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20251212 03:12:24.118837 20961 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 153 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:24.118943 20961 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 153 FOLLOWER]: Advancing to term 154
I20251212 03:12:24.119616 20961 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 154 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:24.119774 20961 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 154 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:24.120026 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 154 candidate_status { last_received { term: 153 index: 125 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:24.120030 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 154 candidate_status { last_received { term: 153 index: 125 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:24.120364 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 154 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:24.120474 20961 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 154 FOLLOWER]: Leader election lost for term 154. Reason: could not achieve majority
I20251212 03:12:24.127292 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 154 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 153. Current term is 154. Ops: []
I20251212 03:12:24.127529 20965 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 153.125, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20251212 03:12:24.127656 20965 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 153 LEADER]: Stepping down as leader of term 153
I20251212 03:12:24.127693 20965 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 153 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:24.127764 20965 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 153.125, Last appended by leader: 125, Current term: 153, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:24.127852 20965 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 153 FOLLOWER]: Advancing to term 154
I20251212 03:12:24.170889 20963 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 153 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:24.170992 20963 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 153 FOLLOWER]: Advancing to term 154
I20251212 03:12:24.171684 20963 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 154 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:24.171844 20963 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 154 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:24.172070 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 154 candidate_status { last_received { term: 153 index: 125 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:24.172088 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 154 candidate_status { last_received { term: 153 index: 125 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:24.172169 20682 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 154 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 154: Already voted for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in this term.
I20251212 03:12:24.172742 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 154 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 154.
I20251212 03:12:24.172941 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 154 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:24.173069 20963 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 154 FOLLOWER]: Leader election won for term 154
I20251212 03:12:24.173149 20963 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 154 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:24.173225 20963 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 153.125, Last appended by leader: 125, Current term: 154, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:24.173691 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 153 to 154, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 154 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:24.242957 20961 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 154 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:24.243063 20961 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 154 FOLLOWER]: Advancing to term 155
I20251212 03:12:24.243860 20961 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 155 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:24.244053 20961 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 155 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:24.244309 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 155 candidate_status { last_received { term: 153 index: 125 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:24.244309 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 155 candidate_status { last_received { term: 153 index: 125 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:24.244458 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 154 FOLLOWER]: Advancing to term 155
I20251212 03:12:24.245213 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 155 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 155.
I20251212 03:12:24.245491 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 155 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:24.245589 20961 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 155 FOLLOWER]: Leader election won for term 155
I20251212 03:12:24.245661 20961 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 155 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:24.245723 20961 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 153.125, Last appended by leader: 125, Current term: 155, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:24.246287 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 154 to 155, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 155 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:24.278033 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 155 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 154. Current term is 155. Ops: []
I20251212 03:12:24.278381 20963 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20251212 03:12:24.278487 20963 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 154 LEADER]: Stepping down as leader of term 154
I20251212 03:12:24.278509 20963 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 154 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:24.278577 20963 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 154.126, Last appended by leader: 126, Current term: 154, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:24.278658 20963 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 154 FOLLOWER]: Advancing to term 155
I20251212 03:12:24.333601 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 155 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 153 index: 125. Preceding OpId from leader: term: 155 index: 126. (index mismatch)
I20251212 03:12:24.333931 20961 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20251212 03:12:24.334960 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 155 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 154 index: 126. Preceding OpId from leader: term: 155 index: 126. (term mismatch)
I20251212 03:12:24.335021 20759 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting all ops after (but not including) 125
I20251212 03:12:24.335042 20759 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting uncommitted NO_OP operation due to leader change: 154.126
I20251212 03:12:24.335062 20759 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 155 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:24.335232 20961 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20251212 03:12:24.506579 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:24.531914 20965 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 155 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:24.532007 20965 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 155 FOLLOWER]: Advancing to term 156
I20251212 03:12:24.532691 20965 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 156 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:24.532842 20965 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 156 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:24.533054 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 156 candidate_status { last_received { term: 155 index: 126 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:24.533054 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 156 candidate_status { last_received { term: 155 index: 126 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:24.533308 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 156 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:24.533435 20965 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 156 FOLLOWER]: Leader election lost for term 156. Reason: could not achieve majority
I20251212 03:12:24.552775 20605 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 156 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 155. Current term is 156. Ops: []
I20251212 03:12:24.553031 20957 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: INVALID_TERM, Last received: 155.126, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20251212 03:12:24.553160 20957 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 155 LEADER]: Stepping down as leader of term 155
I20251212 03:12:24.553193 20957 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 155 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:24.553251 20957 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 155.126, Last appended by leader: 126, Current term: 155, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:24.553332 20957 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 155 FOLLOWER]: Advancing to term 156
I20251212 03:12:24.616997 20963 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 155 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:24.617100 20963 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 155 FOLLOWER]: Advancing to term 156
I20251212 03:12:24.617817 20963 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 156 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:24.618043 20963 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 156 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:24.618276 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 156 candidate_status { last_received { term: 155 index: 126 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:24.618276 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 156 candidate_status { last_received { term: 155 index: 126 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:24.618412 20605 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 156 FOLLOWER]: Leader election vote request: Denying vote to candidate f894b0d567ab4a2ebdcb416d04ef48e5 in current term 156: Already voted for candidate e9d27948dced4eba8f8228ddc4529393 in this term.
I20251212 03:12:24.619016 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 156 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 156.
I20251212 03:12:24.619206 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 156 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:24.619315 20963 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 156 FOLLOWER]: Leader election won for term 156
I20251212 03:12:24.619382 20963 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 156 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:24.619443 20963 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 155.126, Last appended by leader: 126, Current term: 156, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:24.619921 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 155 to 156, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 156 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:24.650807 20965 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 156 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:24.650900 20965 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 156 FOLLOWER]: Advancing to term 157
I20251212 03:12:24.651552 20965 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 157 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:24.651710 20965 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 157 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:24.651921 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 157 candidate_status { last_received { term: 155 index: 126 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:24.651921 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 157 candidate_status { last_received { term: 155 index: 126 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:24.652060 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 156 FOLLOWER]: Advancing to term 157
I20251212 03:12:24.652743 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 157 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 157.
I20251212 03:12:24.652926 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 157 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:24.653024 20965 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 157 FOLLOWER]: Leader election won for term 157
I20251212 03:12:24.653132 20965 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 157 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:24.653218 20965 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 155.126, Last appended by leader: 126, Current term: 157, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:24.653687 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 156 to 157, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 157 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:24.707648 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 157 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 156. Current term is 157. Ops: []
I20251212 03:12:24.708040 20963 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20251212 03:12:24.708173 20963 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 156 LEADER]: Stepping down as leader of term 156
I20251212 03:12:24.708212 20963 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 156 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:24.708290 20963 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 156.127, Last appended by leader: 127, Current term: 156, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:24.708371 20963 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 156 FOLLOWER]: Advancing to term 157
I20251212 03:12:24.753273 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 157 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 156 index: 127. Preceding OpId from leader: term: 157 index: 127. (term mismatch)
I20251212 03:12:24.753281 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 157 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 155 index: 126. Preceding OpId from leader: term: 157 index: 127. (index mismatch)
I20251212 03:12:24.753396 20759 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting all ops after (but not including) 126
I20251212 03:12:24.753443 20759 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting uncommitted NO_OP operation due to leader change: 156.127
I20251212 03:12:24.753466 20759 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 157 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:24.753605 20965 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20251212 03:12:24.753700 20966 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20251212 03:12:25.019989 20963 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 157 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:25.020103 20963 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 157 FOLLOWER]: Advancing to term 158
I20251212 03:12:25.020825 20963 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 158 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:25.020993 20963 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 158 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:25.021235 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 158 candidate_status { last_received { term: 157 index: 127 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:25.021245 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 158 candidate_status { last_received { term: 157 index: 127 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:25.021522 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 158 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:25.021651 20963 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 158 FOLLOWER]: Leader election lost for term 158. Reason: could not achieve majority
I20251212 03:12:25.030242 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 158 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 157. Current term is 158. Ops: []
I20251212 03:12:25.030562 20965 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 157.127, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20251212 03:12:25.030710 20966 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 157 LEADER]: Stepping down as leader of term 157
I20251212 03:12:25.030752 20966 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 157 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:25.030820 20966 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 127, Committed index: 127, Last appended: 157.127, Last appended by leader: 127, Current term: 157, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:25.030910 20966 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 157 FOLLOWER]: Advancing to term 158
I20251212 03:12:25.031638 20957 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 157 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:25.031697 20957 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 157 FOLLOWER]: Advancing to term 158
I20251212 03:12:25.032162 20957 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 158 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:25.032315 20957 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 158 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:25.032465 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 158 candidate_status { last_received { term: 157 index: 127 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:25.032483 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 158 candidate_status { last_received { term: 157 index: 127 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:25.032539 20759 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 158 FOLLOWER]: Leader election vote request: Denying vote to candidate d3fe965dfda94fd4bf3f61a5c0452d63 in current term 158: Already voted for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in this term.
I20251212 03:12:25.033123 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 158 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 158.
I20251212 03:12:25.033381 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 158 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:25.033500 20957 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 158 FOLLOWER]: Leader election won for term 158
I20251212 03:12:25.033566 20957 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 158 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:25.033620 20957 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 127, Committed index: 127, Last appended: 157.127, Last appended by leader: 127, Current term: 158, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:25.034161 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 157 to 158, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 158 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:25.111712 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 158 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 157 index: 127. Preceding OpId from leader: term: 158 index: 128. (index mismatch)
I20251212 03:12:25.112008 20957 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20251212 03:12:25.113129 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 158 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 157 index: 127. Preceding OpId from leader: term: 158 index: 128. (index mismatch)
I20251212 03:12:25.113513 20957 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20251212 03:12:25.214601 20963 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 158 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:25.214706 20963 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 158 FOLLOWER]: Advancing to term 159
I20251212 03:12:25.215404 20963 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 159 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:25.215587 20963 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 159 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:25.215816 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 159 candidate_status { last_received { term: 158 index: 128 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:25.215828 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 159 candidate_status { last_received { term: 158 index: 128 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:25.216066 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 159 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:25.216166 20963 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 159 FOLLOWER]: Leader election lost for term 159. Reason: could not achieve majority
I20251212 03:12:25.239005 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 159 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 158. Current term is 159. Ops: []
I20251212 03:12:25.239413 20967 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 158.128, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20251212 03:12:25.239542 20967 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 158 LEADER]: Stepping down as leader of term 158
I20251212 03:12:25.239579 20967 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 158 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:25.239648 20967 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 158.128, Last appended by leader: 128, Current term: 158, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:25.239730 20967 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 158 FOLLOWER]: Advancing to term 159
I20251212 03:12:25.297596 20966 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 158 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:25.297698 20966 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 158 FOLLOWER]: Advancing to term 159
I20251212 03:12:25.298511 20966 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 159 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:25.298676 20966 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 159 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:25.298888 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 159 candidate_status { last_received { term: 158 index: 128 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:25.298902 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 159 candidate_status { last_received { term: 158 index: 128 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:25.298955 20759 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 159 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 in current term 159: Already voted for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in this term.
I20251212 03:12:25.299549 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 159 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 159.
I20251212 03:12:25.299727 20571 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 159 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:25.299839 20966 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 159 FOLLOWER]: Leader election won for term 159
I20251212 03:12:25.299906 20966 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 159 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:25.299964 20966 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 158.128, Last appended by leader: 128, Current term: 159, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:25.300419 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 158 to 159, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 159 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:25.348600 20963 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 159 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:25.348698 20963 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 159 FOLLOWER]: Advancing to term 160
I20251212 03:12:25.349380 20963 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 160 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:25.349535 20963 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 160 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:25.349761 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 160 candidate_status { last_received { term: 158 index: 128 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:25.349776 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 160 candidate_status { last_received { term: 158 index: 128 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:25.349879 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 159 FOLLOWER]: Advancing to term 160
I20251212 03:12:25.350504 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 160 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 160.
I20251212 03:12:25.350688 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 160 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:25.350791 20963 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 160 FOLLOWER]: Leader election won for term 160
I20251212 03:12:25.350895 20963 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 160 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:25.350939 20963 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 158.128, Last appended by leader: 128, Current term: 160, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:25.351553 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 159 to 160, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 160 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:25.398669 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 160 LEADER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 159. Current term is 160. Ops: []
I20251212 03:12:25.398998 20966 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20251212 03:12:25.399119 20966 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer f894b0d567ab4a2ebdcb416d04ef48e5 log is divergent from this leader: its last log entry 160.129 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 128
I20251212 03:12:25.399168 20966 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 159 LEADER]: Stepping down as leader of term 159
I20251212 03:12:25.399200 20966 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 159 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:25.399260 20966 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 159.129, Last appended by leader: 129, Current term: 159, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:25.399344 20966 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 159 FOLLOWER]: Advancing to term 160
I20251212 03:12:25.450670 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 160 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 158 index: 128. Preceding OpId from leader: term: 160 index: 129. (index mismatch)
I20251212 03:12:25.450949 20963 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20251212 03:12:25.452023 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 160 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 159 index: 129. Preceding OpId from leader: term: 160 index: 129. (term mismatch)
I20251212 03:12:25.452080 20605 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting all ops after (but not including) 128
I20251212 03:12:25.452100 20605 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting uncommitted NO_OP operation due to leader change: 159.129
I20251212 03:12:25.452122 20605 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 160 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:25.452289 20963 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20251212 03:12:25.506811 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:25.552568 20967 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 160 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:25.552711 20967 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 160 FOLLOWER]: Advancing to term 161
I20251212 03:12:25.553457 20967 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 161 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:25.553602 20967 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 161 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:25.553803 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 161 candidate_status { last_received { term: 160 index: 129 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:25.553803 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 161 candidate_status { last_received { term: 160 index: 129 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:25.553922 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 160 FOLLOWER]: Advancing to term 161
I20251212 03:12:25.554512 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 161 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 161.
I20251212 03:12:25.554710 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 161 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, e9d27948dced4eba8f8228ddc4529393; no voters: f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:25.554795 20967 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 161 FOLLOWER]: Leader election won for term 161
I20251212 03:12:25.554848 20967 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 161 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:25.554898 20967 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 160.129, Last appended by leader: 129, Current term: 161, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:25.555356 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 160 to 161, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 161 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:25.571481 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 161 LEADER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 160. Current term is 161. Ops: []
I20251212 03:12:25.571735 20963 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 160.129, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20251212 03:12:25.571826 20963 consensus_queue.cc:1243] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer d3fe965dfda94fd4bf3f61a5c0452d63 log is divergent from this leader: its last log entry 161.130 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 129
I20251212 03:12:25.571882 20968 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 160 LEADER]: Stepping down as leader of term 160
I20251212 03:12:25.571919 20968 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 160 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:25.571977 20968 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 160.129, Last appended by leader: 129, Current term: 160, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:25.572072 20968 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 160 FOLLOWER]: Advancing to term 161
I20251212 03:12:25.639788 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 161 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 160 index: 129. Preceding OpId from leader: term: 161 index: 130. (index mismatch)
I20251212 03:12:25.640105 20967 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20251212 03:12:25.641425 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 161 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 160 index: 129. Preceding OpId from leader: term: 161 index: 130. (index mismatch)
I20251212 03:12:25.641604 20961 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20251212 03:12:25.742172 20966 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 161 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:25.742306 20966 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 161 FOLLOWER]: Advancing to term 162
I20251212 03:12:25.743026 20966 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 162 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:25.743224 20966 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 162 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:25.743475 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 162 candidate_status { last_received { term: 161 index: 130 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:25.743484 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 162 candidate_status { last_received { term: 161 index: 130 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:25.743583 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 161 FOLLOWER]: Advancing to term 162
I20251212 03:12:25.744150 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 162 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 162.
I20251212 03:12:25.744343 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 162 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:25.744442 20966 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 162 FOLLOWER]: Leader election won for term 162
I20251212 03:12:25.744562 20966 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 162 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:25.744656 20966 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 130, Committed index: 130, Last appended: 161.130, Last appended by leader: 130, Current term: 162, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:25.745182 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 161 to 162, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 162 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:25.759430 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 162 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 161. Current term is 162. Ops: []
I20251212 03:12:25.759686 20957 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 161.130, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20251212 03:12:25.759797 20957 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 161 LEADER]: Stepping down as leader of term 161
I20251212 03:12:25.759828 20957 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 161 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:25.759896 20957 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 130, Committed index: 130, Last appended: 161.130, Last appended by leader: 130, Current term: 161, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:25.759974 20957 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 161 FOLLOWER]: Advancing to term 162
I20251212 03:12:25.824422 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 162 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 161 index: 130. Preceding OpId from leader: term: 162 index: 131. (index mismatch)
I20251212 03:12:25.824702 20966 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20251212 03:12:25.825757 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 162 FOLLOWER]: Refusing update from remote peer e9d27948dced4eba8f8228ddc4529393: Log matching property violated. Preceding OpId in replica: term: 161 index: 130. Preceding OpId from leader: term: 162 index: 131. (index mismatch)
I20251212 03:12:25.825963 20966 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20251212 03:12:25.926175 20968 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 162 FOLLOWER]: Starting leader election (detected failure of leader e9d27948dced4eba8f8228ddc4529393)
I20251212 03:12:25.926311 20968 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 162 FOLLOWER]: Advancing to term 163
I20251212 03:12:25.927327 20968 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 163 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:25.927557 20968 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 163 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:25.927767 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 163 candidate_status { last_received { term: 162 index: 131 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:25.927778 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 163 candidate_status { last_received { term: 162 index: 131 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:25.927875 20682 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 162 FOLLOWER]: Advancing to term 163
I20251212 03:12:25.928623 20682 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 163 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 163.
I20251212 03:12:25.928808 20725 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 163 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:25.928902 20968 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 163 FOLLOWER]: Leader election won for term 163
I20251212 03:12:25.928973 20968 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 163 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:25.929025 20968 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 131, Committed index: 131, Last appended: 162.131, Last appended by leader: 131, Current term: 163, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:25.929592 20498 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 162 to 163, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 163 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:25.933080 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 163 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 162. Current term is 163. Ops: []
I20251212 03:12:25.933414 20969 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 162.131, Next index: 132, Last known committed idx: 131, Time since last communication: 0.000s
I20251212 03:12:25.933529 20969 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 162 LEADER]: Stepping down as leader of term 162
I20251212 03:12:25.933566 20969 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 162 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:25.933630 20969 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 131, Committed index: 131, Last appended: 162.131, Last appended by leader: 131, Current term: 162, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:25.933712 20969 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 162 FOLLOWER]: Advancing to term 163
I20251212 03:12:26.012030 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 163 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 162 index: 131. Preceding OpId from leader: term: 163 index: 132. (index mismatch)
I20251212 03:12:26.012333 20968 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 132, Last known committed idx: 131, Time since last communication: 0.000s
I20251212 03:12:26.013438 20682 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 163 FOLLOWER]: Refusing update from remote peer f894b0d567ab4a2ebdcb416d04ef48e5: Log matching property violated. Preceding OpId in replica: term: 162 index: 131. Preceding OpId from leader: term: 163 index: 132. (index mismatch)
I20251212 03:12:26.013638 20968 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 132, Last known committed idx: 131, Time since last communication: 0.000s
I20251212 03:12:26.114814 20957 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 163 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:26.114930 20957 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 163 FOLLOWER]: Advancing to term 164
I20251212 03:12:26.115597 20957 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 164 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:26.115765 20957 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 164 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:26.115973 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 164 candidate_status { last_received { term: 163 index: 132 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:26.115983 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 164 candidate_status { last_received { term: 163 index: 132 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:26.116257 20646 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 164 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63; no voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5
I20251212 03:12:26.116379 20957 raft_consensus.cc:2749] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 164 FOLLOWER]: Leader election lost for term 164. Reason: could not achieve majority
I20251212 03:12:26.121554 20682 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 164 FOLLOWER]: Rejecting Update request from peer f894b0d567ab4a2ebdcb416d04ef48e5 for earlier term 163. Current term is 164. Ops: []
I20251212 03:12:26.121834 20970 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 }, Status: INVALID_TERM, Last received: 163.132, Next index: 133, Last known committed idx: 132, Time since last communication: 0.000s
I20251212 03:12:26.121966 20970 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 163 LEADER]: Stepping down as leader of term 163
I20251212 03:12:26.121995 20970 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 163 LEADER]: Becoming Follower/Learner. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:26.122059 20970 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 132, Committed index: 132, Last appended: 163.132, Last appended by leader: 132, Current term: 163, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:26.122298 20970 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 163 FOLLOWER]: Advancing to term 164
I20251212 03:12:26.204041 20969 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 163 FOLLOWER]: Starting leader election (detected failure of leader f894b0d567ab4a2ebdcb416d04ef48e5)
I20251212 03:12:26.204147 20969 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 163 FOLLOWER]: Advancing to term 164
I20251212 03:12:26.204926 20969 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 164 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:26.205137 20969 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 164 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:26.205322 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 164 candidate_status { last_received { term: 163 index: 132 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:26.205329 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "e9d27948dced4eba8f8228ddc4529393" candidate_term: 164 candidate_status { last_received { term: 163 index: 132 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:26.205426 20682 raft_consensus.cc:2393] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 164 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d27948dced4eba8f8228ddc4529393 in current term 164: Already voted for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in this term.
I20251212 03:12:26.205988 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 164 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d27948dced4eba8f8228ddc4529393 in term 164.
I20251212 03:12:26.206259 20568 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [CANDIDATE]: Term 164 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:26.206373 20969 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 164 FOLLOWER]: Leader election won for term 164
I20251212 03:12:26.206444 20969 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 164 LEADER]: Becoming Leader. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:26.206507 20969 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 132, Committed index: 132, Last appended: 163.132, Last appended by leader: 132, Current term: 164, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:26.207005 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 reported cstate change: term changed from 163 to 164, leader changed from f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195) to e9d27948dced4eba8f8228ddc4529393 (127.19.253.193). New cstate: current_term: 164 leader_uuid: "e9d27948dced4eba8f8228ddc4529393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:26.264900 20957 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 164 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:26.265007 20957 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 164 FOLLOWER]: Advancing to term 165
I20251212 03:12:26.265699 20957 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 165 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:26.265863 20957 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 165 election: Requested vote from peers f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195:46559), e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365)
I20251212 03:12:26.266116 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 165 candidate_status { last_received { term: 163 index: 132 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:26.266240 20759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" candidate_term: 165 candidate_status { last_received { term: 163 index: 132 } } ignore_live_leader: false dest_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5"
I20251212 03:12:26.266333 20759 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 164 FOLLOWER]: Advancing to term 165
I20251212 03:12:26.266920 20759 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 165 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3fe965dfda94fd4bf3f61a5c0452d63 in term 165.
I20251212 03:12:26.267093 20645 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [CANDIDATE]: Term 165 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3fe965dfda94fd4bf3f61a5c0452d63, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: e9d27948dced4eba8f8228ddc4529393
I20251212 03:12:26.267318 20957 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 165 FOLLOWER]: Leader election won for term 165
I20251212 03:12:26.267395 20957 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 165 LEADER]: Becoming Leader. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:26.267437 20957 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 132, Committed index: 132, Last appended: 163.132, Last appended by leader: 132, Current term: 165, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:26.267972 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 reported cstate change: term changed from 164 to 165, leader changed from e9d27948dced4eba8f8228ddc4529393 (127.19.253.193) to d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194). New cstate: current_term: 165 leader_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:26.305869 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 165 FOLLOWER]: Rejecting Update request from peer e9d27948dced4eba8f8228ddc4529393 for earlier term 164. Current term is 165. Ops: []
I20251212 03:12:26.306250 20969 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 0.0, Next index: 133, Last known committed idx: 132, Time since last communication: 0.000s
I20251212 03:12:26.306396 20969 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 164 LEADER]: Stepping down as leader of term 164
I20251212 03:12:26.306435 20969 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 164 LEADER]: Becoming Follower/Learner. State: Replica: e9d27948dced4eba8f8228ddc4529393, State: Running, Role: LEADER
I20251212 03:12:26.306494 20969 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 132, Committed index: 132, Last appended: 164.133, Last appended by leader: 133, Current term: 164, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:26.306582 20969 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 164 FOLLOWER]: Advancing to term 165
I20251212 03:12:26.346304 20759 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 165 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 163 index: 132. Preceding OpId from leader: term: 165 index: 133. (index mismatch)
I20251212 03:12:26.346728 20957 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 133, Last known committed idx: 132, Time since last communication: 0.000s
I20251212 03:12:26.347877 20605 raft_consensus.cc:1275] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 165 FOLLOWER]: Refusing update from remote peer d3fe965dfda94fd4bf3f61a5c0452d63: Log matching property violated. Preceding OpId in replica: term: 164 index: 133. Preceding OpId from leader: term: 165 index: 133. (term mismatch)
I20251212 03:12:26.347946 20605 pending_rounds.cc:85] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting all ops after (but not including) 132
I20251212 03:12:26.347973 20605 pending_rounds.cc:107] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: Aborting uncommitted NO_OP operation due to leader change: 164.133
I20251212 03:12:26.347997 20605 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 165 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:26.348201 20971 consensus_queue.cc:1048] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 133, Last known committed idx: 132, Time since last communication: 0.000s
I20251212 03:12:26.448925 20970 raft_consensus.cc:493] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 165 FOLLOWER]: Starting leader election (detected failure of leader d3fe965dfda94fd4bf3f61a5c0452d63)
I20251212 03:12:26.449038 20970 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 165 FOLLOWER]: Advancing to term 166
I20251212 03:12:26.450047 20970 raft_consensus.cc:515] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 166 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:26.450182 20759 raft_consensus.cc:1240] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 166 FOLLOWER]: Rejecting Update request from peer d3fe965dfda94fd4bf3f61a5c0452d63 for earlier term 165. Current term is 166. Ops: []
I20251212 03:12:26.450385 20970 leader_election.cc:290] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 166 election: Requested vote from peers e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365), d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515)
I20251212 03:12:26.450410 20605 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 166 candidate_status { last_received { term: 165 index: 133 } } ignore_live_leader: false dest_uuid: "e9d27948dced4eba8f8228ddc4529393"
I20251212 03:12:26.450513 20605 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 165 FOLLOWER]: Advancing to term 166
I20251212 03:12:26.450564 20971 consensus_queue.cc:1059] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 }, Status: INVALID_TERM, Last received: 165.133, Next index: 134, Last known committed idx: 133, Time since last communication: 0.000s
I20251212 03:12:26.450564 20682 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d360069b7514dfab215af64b6929a16" candidate_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" candidate_term: 166 candidate_status { last_received { term: 165 index: 133 } } ignore_live_leader: false dest_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63"
I20251212 03:12:26.450685 20971 raft_consensus.cc:3055] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 165 LEADER]: Stepping down as leader of term 165
I20251212 03:12:26.450731 20971 raft_consensus.cc:740] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 165 LEADER]: Becoming Follower/Learner. State: Replica: d3fe965dfda94fd4bf3f61a5c0452d63, State: Running, Role: LEADER
I20251212 03:12:26.450784 20971 consensus_queue.cc:260] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 133, Committed index: 133, Last appended: 165.133, Last appended by leader: 133, Current term: 165, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:26.450860 20971 raft_consensus.cc:3060] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 165 FOLLOWER]: Advancing to term 166
I20251212 03:12:26.451287 20605 raft_consensus.cc:2468] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 166 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f894b0d567ab4a2ebdcb416d04ef48e5 in term 166.
I20251212 03:12:26.451470 20723 leader_election.cc:304] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [CANDIDATE]: Term 166 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9d27948dced4eba8f8228ddc4529393, f894b0d567ab4a2ebdcb416d04ef48e5; no voters: d3fe965dfda94fd4bf3f61a5c0452d63
I20251212 03:12:26.451576 20970 raft_consensus.cc:2804] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 166 FOLLOWER]: Leader election won for term 166
I20251212 03:12:26.451638 20970 raft_consensus.cc:697] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 166 LEADER]: Becoming Leader. State: Replica: f894b0d567ab4a2ebdcb416d04ef48e5, State: Running, Role: LEADER
I20251212 03:12:26.451694 20970 consensus_queue.cc:237] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 133, Committed index: 133, Last appended: 165.133, Last appended by leader: 133, Current term: 166, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } }
I20251212 03:12:26.452189 20501 catalog_manager.cc:5654] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 reported cstate change: term changed from 165 to 166, leader changed from d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194) to f894b0d567ab4a2ebdcb416d04ef48e5 (127.19.253.195). New cstate: current_term: 166 leader_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f894b0d567ab4a2ebdcb416d04ef48e5" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9d27948dced4eba8f8228ddc4529393" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3fe965dfda94fd4bf3f61a5c0452d63" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:26.507071 20471 timestamp_advancement-itest.cc:267] GCing logs...
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/timestamp_advancement-itest.cc:267: Failure
Expected: (gcable_size) > (0), actual: 0 vs 0
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_util.cc:403: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/timestamp_advancement-itest.cc:332: Failure
Expected: GCUntilNoWritesInWAL(ts, replica) doesn't generate new fatal failures in the current thread.
Actual: it does.
I20251212 03:12:26.507369 20471 tablet_server.cc:178] TabletServer@127.19.253.193:0 shutting down...
I20251212 03:12:26.523483 20471 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251212 03:12:26.523676 20471 tablet_replica.cc:333] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393: stopping tablet replica
I20251212 03:12:26.523772 20471 raft_consensus.cc:2243] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 166 FOLLOWER]: Raft consensus shutting down.
I20251212 03:12:26.523845 20471 raft_consensus.cc:2272] T 3d360069b7514dfab215af64b6929a16 P e9d27948dced4eba8f8228ddc4529393 [term 166 FOLLOWER]: Raft consensus is shut down!
I20251212 03:12:26.536689 20471 tablet_server.cc:195] TabletServer@127.19.253.193:0 shutdown complete.
I20251212 03:12:26.538101 20471 tablet_server.cc:178] TabletServer@127.19.253.194:0 shutting down...
W20251212 03:12:26.551674 20725 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251212 03:12:26.552058 20725 consensus_peers.cc:597] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 -> Peer d3fe965dfda94fd4bf3f61a5c0452d63 (127.19.253.194:45515): Couldn't send request to peer d3fe965dfda94fd4bf3f61a5c0452d63. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20251212 03:12:26.560043 20471 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251212 03:12:26.560185 20471 tablet_replica.cc:333] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63: stopping tablet replica
I20251212 03:12:26.560338 20471 raft_consensus.cc:2243] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 166 FOLLOWER]: Raft consensus shutting down.
I20251212 03:12:26.560425 20471 raft_consensus.cc:2272] T 3d360069b7514dfab215af64b6929a16 P d3fe965dfda94fd4bf3f61a5c0452d63 [term 166 FOLLOWER]: Raft consensus is shut down!
W20251212 03:12:26.573053 20723 consensus_peers.cc:597] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 -> Peer e9d27948dced4eba8f8228ddc4529393 (127.19.253.193:46365): Couldn't send request to peer e9d27948dced4eba8f8228ddc4529393. Status: Network error: Client connection negotiation failed: client connection to 127.19.253.193:46365: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251212 03:12:26.573297 20471 tablet_server.cc:195] TabletServer@127.19.253.194:0 shutdown complete.
I20251212 03:12:26.574420 20471 tablet_server.cc:178] TabletServer@127.19.253.195:0 shutting down...
I20251212 03:12:26.828107 20471 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251212 03:12:26.828313 20471 tablet_replica.cc:333] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: stopping tablet replica
I20251212 03:12:26.828387 20471 raft_consensus.cc:2243] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 166 LEADER]: Raft consensus shutting down.
I20251212 03:12:26.828473 20471 pending_rounds.cc:70] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Trying to abort 1 pending ops.
I20251212 03:12:26.828495 20471 pending_rounds.cc:77] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5: Aborting op as it isn't in flight: id { term: 166 index: 134 } timestamp: 7231525463866593280 op_type: NO_OP noop_request { }
I20251212 03:12:26.828555 20471 raft_consensus.cc:2889] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 166 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251212 03:12:26.828573 20471 raft_consensus.cc:2272] T 3d360069b7514dfab215af64b6929a16 P f894b0d567ab4a2ebdcb416d04ef48e5 [term 166 FOLLOWER]: Raft consensus is shut down!
I20251212 03:12:26.840910 20471 tablet_server.cc:195] TabletServer@127.19.253.195:0 shutdown complete.
I20251212 03:12:26.841961 20471 master.cc:561] Master@127.19.253.254:39173 shutting down...
I20251212 03:12:26.952309 20471 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2 [term 1 LEADER]: Raft consensus shutting down.
I20251212 03:12:26.952433 20471 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251212 03:12:26.952482 20471 tablet_replica.cc:333] T 00000000000000000000000000000000 P 93079d4189264fb3ba14dc72752898c2: stopping tablet replica
I20251212 03:12:26.964650 20471 master.cc:583] Master@127.19.253.254:39173 shutdown complete.
I20251212 03:12:26.966715 20471 test_util.cc:183] -----------------------------------------------
I20251212 03:12:26.966771 20471 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1765509114134682-20471-0
[ FAILED ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData (32827 ms)
[ RUN ] TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap
I20251212 03:12:26.967209 20471 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.253.254:39701
I20251212 03:12:26.967348 20471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 03:12:26.968720 20977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:26.968745 20974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:26.968806 20975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:12:26.968891 20471 server_base.cc:1047] running on GCE node
I20251212 03:12:26.969095 20471 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251212 03:12:26.969131 20471 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251212 03:12:26.969153 20471 hybrid_clock.cc:648] HybridClock initialized: now 1765509146969153 us; error 0 us; skew 500 ppm
I20251212 03:12:26.969731 20471 webserver.cc:492] Webserver started at http://127.19.253.254:45153/ using document root <none> and password file <none>
I20251212 03:12:26.969825 20471 fs_manager.cc:362] Metadata directory not provided
I20251212 03:12:26.969878 20471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 03:12:26.969928 20471 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251212 03:12:26.970187 20471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765509114134682-20471-0/minicluster-data/master-0-root/instance:
uuid: "8e76e3d2bb9442aebce83fe50ed3349b"
format_stamp: "Formatted at 2025-12-12 03:12:26 on dist-test-slave-q05w"
I20251212 03:12:26.971086 20471 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251212 03:12:26.971583 20982 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:26.971732 20471 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:26.971781 20471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765509114134682-20471-0/minicluster-data/master-0-root
uuid: "8e76e3d2bb9442aebce83fe50ed3349b"
format_stamp: "Formatted at 2025-12-12 03:12:26 on dist-test-slave-q05w"
I20251212 03:12:26.971827 20471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765509114134682-20471-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765509114134682-20471-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765509114134682-20471-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 03:12:27.001922 20471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 03:12:27.002147 20471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 03:12:27.004805 20471 rpc_server.cc:307] RPC server started. Bound to: 127.19.253.254:39701
I20251212 03:12:27.006759 21034 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.253.254:39701 every 8 connection(s)
I20251212 03:12:27.006901 21035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 03:12:27.007922 21035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e76e3d2bb9442aebce83fe50ed3349b: Bootstrap starting.
I20251212 03:12:27.008203 21035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e76e3d2bb9442aebce83fe50ed3349b: Neither blocks nor log segments found. Creating new log.
I20251212 03:12:27.008821 21035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e76e3d2bb9442aebce83fe50ed3349b: No bootstrap required, opened a new log
I20251212 03:12:27.008950 21035 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8e76e3d2bb9442aebce83fe50ed3349b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e76e3d2bb9442aebce83fe50ed3349b" member_type: VOTER }
I20251212 03:12:27.009001 21035 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8e76e3d2bb9442aebce83fe50ed3349b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 03:12:27.009016 21035 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8e76e3d2bb9442aebce83fe50ed3349b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e76e3d2bb9442aebce83fe50ed3349b, State: Initialized, Role: FOLLOWER
I20251212 03:12:27.009054 21035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e76e3d2bb9442aebce83fe50ed3349b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e76e3d2bb9442aebce83fe50ed3349b" member_type: VOTER }
I20251212 03:12:27.009104 21035 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8e76e3d2bb9442aebce83fe50ed3349b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251212 03:12:27.009130 21035 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8e76e3d2bb9442aebce83fe50ed3349b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251212 03:12:27.009155 21035 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8e76e3d2bb9442aebce83fe50ed3349b [term 0 FOLLOWER]: Advancing to term 1
I20251212 03:12:27.009707 21035 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8e76e3d2bb9442aebce83fe50ed3349b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e76e3d2bb9442aebce83fe50ed3349b" member_type: VOTER }
I20251212 03:12:27.009771 21035 leader_election.cc:304] T 00000000000000000000000000000000 P 8e76e3d2bb9442aebce83fe50ed3349b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e76e3d2bb9442aebce83fe50ed3349b; no voters:
I20251212 03:12:27.009861 21035 leader_election.cc:290] T 00000000000000000000000000000000 P 8e76e3d2bb9442aebce83fe50ed3349b [CANDIDATE]: Term 1 election: Requested vote from peers
I20251212 03:12:27.009927 21038 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8e76e3d2bb9442aebce83fe50ed3349b [term 1 FOLLOWER]: Leader election won for term 1
I20251212 03:12:27.010010 21035 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8e76e3d2bb9442aebce83fe50ed3349b [sys.catalog]: configured and running, proceeding with master startup.
I20251212 03:12:27.010084 21038 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8e76e3d2bb9442aebce83fe50ed3349b [term 1 LEADER]: Becoming Leader. State: Replica: 8e76e3d2bb9442aebce83fe50ed3349b, State: Running, Role: LEADER
I20251212 03:12:27.010143 21038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e76e3d2bb9442aebce83fe50ed3349b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e76e3d2bb9442aebce83fe50ed3349b" member_type: VOTER }
I20251212 03:12:27.010357 21038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e76e3d2bb9442aebce83fe50ed3349b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e76e3d2bb9442aebce83fe50ed3349b. Latest consensus state: current_term: 1 leader_uuid: "8e76e3d2bb9442aebce83fe50ed3349b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e76e3d2bb9442aebce83fe50ed3349b" member_type: VOTER } }
I20251212 03:12:27.010399 21040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e76e3d2bb9442aebce83fe50ed3349b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e76e3d2bb9442aebce83fe50ed3349b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e76e3d2bb9442aebce83fe50ed3349b" member_type: VOTER } }
I20251212 03:12:27.010591 21040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e76e3d2bb9442aebce83fe50ed3349b [sys.catalog]: This master's current role is: LEADER
I20251212 03:12:27.010499 21038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e76e3d2bb9442aebce83fe50ed3349b [sys.catalog]: This master's current role is: LEADER
I20251212 03:12:27.010860 21046 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251212 03:12:27.011049 21046 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251212 03:12:27.011420 20471 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251212 03:12:27.011644 21046 catalog_manager.cc:1357] Generated new cluster ID: 054980c3a94b43a39604e35a2037d369
I20251212 03:12:27.011691 21046 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251212 03:12:27.020771 21046 catalog_manager.cc:1380] Generated new certificate authority record
I20251212 03:12:27.021243 21046 catalog_manager.cc:1514] Loading token signing keys...
I20251212 03:12:27.027706 21046 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 8e76e3d2bb9442aebce83fe50ed3349b: Generated new TSK 0
I20251212 03:12:27.027803 21046 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251212 03:12:27.043444 20471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 03:12:27.044977 21056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:27.045118 21059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:27.045035 21057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:12:27.045106 20471 server_base.cc:1047] running on GCE node
I20251212 03:12:27.045346 20471 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251212 03:12:27.045387 20471 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251212 03:12:27.045401 20471 hybrid_clock.cc:648] HybridClock initialized: now 1765509147045401 us; error 0 us; skew 500 ppm
I20251212 03:12:27.045972 20471 webserver.cc:492] Webserver started at http://127.19.253.193:46603/ using document root <none> and password file <none>
I20251212 03:12:27.046072 20471 fs_manager.cc:362] Metadata directory not provided
I20251212 03:12:27.046135 20471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 03:12:27.046228 20471 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251212 03:12:27.046517 20471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765509114134682-20471-0/minicluster-data/ts-0-root/instance:
uuid: "95ad0197220142e8ae9cb7546f4a2bcd"
format_stamp: "Formatted at 2025-12-12 03:12:27 on dist-test-slave-q05w"
I20251212 03:12:27.047482 20471 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251212 03:12:27.047977 21064 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:27.048118 20471 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251212 03:12:27.048203 20471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765509114134682-20471-0/minicluster-data/ts-0-root
uuid: "95ad0197220142e8ae9cb7546f4a2bcd"
format_stamp: "Formatted at 2025-12-12 03:12:27 on dist-test-slave-q05w"
I20251212 03:12:27.048259 20471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765509114134682-20471-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765509114134682-20471-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765509114134682-20471-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 03:12:27.060403 20471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 03:12:27.060637 20471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 03:12:27.060801 20471 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251212 03:12:27.061007 20471 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251212 03:12:27.061034 20471 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:27.061069 20471 ts_tablet_manager.cc:616] Registered 0 tablets
I20251212 03:12:27.061098 20471 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:27.063977 20471 rpc_server.cc:307] RPC server started. Bound to: 127.19.253.193:36423
I20251212 03:12:27.064038 21127 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.253.193:36423 every 8 connection(s)
I20251212 03:12:27.064347 20471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 03:12:27.065659 21133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:27.065717 21132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:27.065917 21135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:12:27.066042 20471 server_base.cc:1047] running on GCE node
I20251212 03:12:27.066167 20471 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251212 03:12:27.066195 20471 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251212 03:12:27.066239 20471 hybrid_clock.cc:648] HybridClock initialized: now 1765509147066238 us; error 0 us; skew 500 ppm
I20251212 03:12:27.066891 20471 webserver.cc:492] Webserver started at http://127.19.253.194:38449/ using document root <none> and password file <none>
I20251212 03:12:27.066995 20471 fs_manager.cc:362] Metadata directory not provided
I20251212 03:12:27.067041 20471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 03:12:27.067087 20471 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251212 03:12:27.067438 20471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765509114134682-20471-0/minicluster-data/ts-1-root/instance:
uuid: "80f56db926b8409480a8452b3a121f58"
format_stamp: "Formatted at 2025-12-12 03:12:27 on dist-test-slave-q05w"
I20251212 03:12:27.068346 20471 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251212 03:12:27.068861 21140 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:27.069001 20471 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251212 03:12:27.069056 20471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765509114134682-20471-0/minicluster-data/ts-1-root
uuid: "80f56db926b8409480a8452b3a121f58"
format_stamp: "Formatted at 2025-12-12 03:12:27 on dist-test-slave-q05w"
I20251212 03:12:27.069103 20471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765509114134682-20471-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765509114134682-20471-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765509114134682-20471-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 03:12:27.072934 21128 heartbeater.cc:344] Connected to a master server at 127.19.253.254:39701
I20251212 03:12:27.072995 21128 heartbeater.cc:461] Registering TS with master...
I20251212 03:12:27.073107 21128 heartbeater.cc:507] Master 127.19.253.254:39701 requested a full tablet report, sending...
I20251212 03:12:27.073354 20999 ts_manager.cc:194] Registered new tserver with Master: 95ad0197220142e8ae9cb7546f4a2bcd (127.19.253.193:36423)
I20251212 03:12:27.073920 20999 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47932
I20251212 03:12:27.087394 20471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 03:12:27.087611 20471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 03:12:27.087787 20471 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251212 03:12:27.087982 20471 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251212 03:12:27.088011 20471 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:27.088045 20471 ts_tablet_manager.cc:616] Registered 0 tablets
I20251212 03:12:27.088073 20471 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:27.091117 20471 rpc_server.cc:307] RPC server started. Bound to: 127.19.253.194:33281
I20251212 03:12:27.091197 21203 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.253.194:33281 every 8 connection(s)
I20251212 03:12:27.091504 20471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 03:12:27.092970 21207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:27.093178 21210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:12:27.093874 20471 server_base.cc:1047] running on GCE node
W20251212 03:12:27.093911 21208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:12:27.094187 20471 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251212 03:12:27.094350 20471 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251212 03:12:27.094388 20471 hybrid_clock.cc:648] HybridClock initialized: now 1765509147094387 us; error 0 us; skew 500 ppm
I20251212 03:12:27.094679 21204 heartbeater.cc:344] Connected to a master server at 127.19.253.254:39701
I20251212 03:12:27.094725 21204 heartbeater.cc:461] Registering TS with master...
I20251212 03:12:27.094823 21204 heartbeater.cc:507] Master 127.19.253.254:39701 requested a full tablet report, sending...
I20251212 03:12:27.095144 20999 ts_manager.cc:194] Registered new tserver with Master: 80f56db926b8409480a8452b3a121f58 (127.19.253.194:33281)
I20251212 03:12:27.095235 20471 webserver.cc:492] Webserver started at http://127.19.253.195:36055/ using document root <none> and password file <none>
I20251212 03:12:27.095330 20471 fs_manager.cc:362] Metadata directory not provided
I20251212 03:12:27.095367 20471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 03:12:27.095417 20471 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251212 03:12:27.095654 20999 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47940
I20251212 03:12:27.095697 20471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765509114134682-20471-0/minicluster-data/ts-2-root/instance:
uuid: "8ac58b6f4a0b43eaad688ad29d00523c"
format_stamp: "Formatted at 2025-12-12 03:12:27 on dist-test-slave-q05w"
I20251212 03:12:27.096681 20471 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251212 03:12:27.097208 21215 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:27.097393 20471 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251212 03:12:27.097465 20471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765509114134682-20471-0/minicluster-data/ts-2-root
uuid: "8ac58b6f4a0b43eaad688ad29d00523c"
format_stamp: "Formatted at 2025-12-12 03:12:27 on dist-test-slave-q05w"
I20251212 03:12:27.097512 20471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765509114134682-20471-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765509114134682-20471-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765509114134682-20471-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 03:12:27.103694 20471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 03:12:27.103948 20471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 03:12:27.104117 20471 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251212 03:12:27.104384 20471 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251212 03:12:27.104418 20471 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:27.104450 20471 ts_tablet_manager.cc:616] Registered 0 tablets
I20251212 03:12:27.104473 20471 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:27.107942 20471 rpc_server.cc:307] RPC server started. Bound to: 127.19.253.195:40203
I20251212 03:12:27.107975 21278 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.253.195:40203 every 8 connection(s)
I20251212 03:12:27.111474 21279 heartbeater.cc:344] Connected to a master server at 127.19.253.254:39701
I20251212 03:12:27.111538 21279 heartbeater.cc:461] Registering TS with master...
I20251212 03:12:27.111625 21279 heartbeater.cc:507] Master 127.19.253.254:39701 requested a full tablet report, sending...
I20251212 03:12:27.111850 20999 ts_manager.cc:194] Registered new tserver with Master: 8ac58b6f4a0b43eaad688ad29d00523c (127.19.253.195:40203)
I20251212 03:12:27.112176 20999 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47950
I20251212 03:12:27.112519 20471 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004416118s
I20251212 03:12:27.120436 20471 test_util.cc:276] Using random seed: -1584409806
I20251212 03:12:27.125371 20999 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47978:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251212 03:12:27.125501 20999 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251212 03:12:27.130385 21243 tablet_service.cc:1505] Processing CreateTablet for tablet bc80f8d16dd043b187b09967fc93dbe3 (DEFAULT_TABLE table=test-workload [id=f0ae6bfdf4224bc7ad7ac56b052cb55a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 03:12:27.130546 21243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc80f8d16dd043b187b09967fc93dbe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 03:12:27.130631 21092 tablet_service.cc:1505] Processing CreateTablet for tablet bc80f8d16dd043b187b09967fc93dbe3 (DEFAULT_TABLE table=test-workload [id=f0ae6bfdf4224bc7ad7ac56b052cb55a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 03:12:27.130714 21092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc80f8d16dd043b187b09967fc93dbe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 03:12:27.132038 21300 tablet_bootstrap.cc:492] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd: Bootstrap starting.
I20251212 03:12:27.132203 21301 tablet_bootstrap.cc:492] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c: Bootstrap starting.
I20251212 03:12:27.132583 21300 tablet_bootstrap.cc:654] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd: Neither blocks nor log segments found. Creating new log.
I20251212 03:12:27.133005 21301 tablet_bootstrap.cc:654] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c: Neither blocks nor log segments found. Creating new log.
I20251212 03:12:27.133127 21300 tablet_bootstrap.cc:492] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd: No bootstrap required, opened a new log
I20251212 03:12:27.133183 21300 ts_tablet_manager.cc:1403] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251212 03:12:27.133402 21300 raft_consensus.cc:359] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:27.133482 21300 raft_consensus.cc:385] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 03:12:27.133513 21300 raft_consensus.cc:740] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95ad0197220142e8ae9cb7546f4a2bcd, State: Initialized, Role: FOLLOWER
I20251212 03:12:27.133543 21168 tablet_service.cc:1505] Processing CreateTablet for tablet bc80f8d16dd043b187b09967fc93dbe3 (DEFAULT_TABLE table=test-workload [id=f0ae6bfdf4224bc7ad7ac56b052cb55a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 03:12:27.133562 21300 consensus_queue.cc:260] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:27.133641 21168 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc80f8d16dd043b187b09967fc93dbe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 03:12:27.133738 21300 ts_tablet_manager.cc:1434] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:27.133762 21128 heartbeater.cc:499] Master 127.19.253.254:39701 was elected leader, sending a full tablet report...
I20251212 03:12:27.134021 21301 tablet_bootstrap.cc:492] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c: No bootstrap required, opened a new log
I20251212 03:12:27.134070 21301 ts_tablet_manager.cc:1403] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251212 03:12:27.134225 21301 raft_consensus.cc:359] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:27.134294 21301 raft_consensus.cc:385] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 03:12:27.134349 21301 raft_consensus.cc:740] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac58b6f4a0b43eaad688ad29d00523c, State: Initialized, Role: FOLLOWER
I20251212 03:12:27.134419 21301 consensus_queue.cc:260] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:27.134625 21301 ts_tablet_manager.cc:1434] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251212 03:12:27.134649 21279 heartbeater.cc:499] Master 127.19.253.254:39701 was elected leader, sending a full tablet report...
I20251212 03:12:27.135306 21306 tablet_bootstrap.cc:492] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58: Bootstrap starting.
I20251212 03:12:27.135721 21306 tablet_bootstrap.cc:654] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58: Neither blocks nor log segments found. Creating new log.
I20251212 03:12:27.136214 21306 tablet_bootstrap.cc:492] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58: No bootstrap required, opened a new log
I20251212 03:12:27.136255 21306 ts_tablet_manager.cc:1403] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251212 03:12:27.136365 21306 raft_consensus.cc:359] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:27.136407 21306 raft_consensus.cc:385] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 03:12:27.136420 21306 raft_consensus.cc:740] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80f56db926b8409480a8452b3a121f58, State: Initialized, Role: FOLLOWER
I20251212 03:12:27.136456 21306 consensus_queue.cc:260] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:27.136613 21306 ts_tablet_manager.cc:1434] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:27.136656 21204 heartbeater.cc:499] Master 127.19.253.254:39701 was elected leader, sending a full tablet report...
I20251212 03:12:27.139721 21305 raft_consensus.cc:493] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 03:12:27.139792 21305 raft_consensus.cc:515] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:27.139989 21305 leader_election.cc:290] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95ad0197220142e8ae9cb7546f4a2bcd (127.19.253.193:36423), 80f56db926b8409480a8452b3a121f58 (127.19.253.194:33281)
I20251212 03:12:27.142577 21178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80f56db926b8409480a8452b3a121f58" is_pre_election: true
I20251212 03:12:27.142577 21102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" is_pre_election: true
I20251212 03:12:27.142691 21178 raft_consensus.cc:2468] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ac58b6f4a0b43eaad688ad29d00523c in term 0.
I20251212 03:12:27.142694 21102 raft_consensus.cc:2468] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ac58b6f4a0b43eaad688ad29d00523c in term 0.
I20251212 03:12:27.142858 21219 leader_election.cc:304] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac58b6f4a0b43eaad688ad29d00523c, 95ad0197220142e8ae9cb7546f4a2bcd; no voters:
I20251212 03:12:27.142954 21305 raft_consensus.cc:2804] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251212 03:12:27.142990 21305 raft_consensus.cc:493] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:27.143018 21305 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 0 FOLLOWER]: Advancing to term 1
I20251212 03:12:27.143584 21305 raft_consensus.cc:515] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:27.143699 21305 leader_election.cc:290] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [CANDIDATE]: Term 1 election: Requested vote from peers 95ad0197220142e8ae9cb7546f4a2bcd (127.19.253.193:36423), 80f56db926b8409480a8452b3a121f58 (127.19.253.194:33281)
I20251212 03:12:27.143838 21178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80f56db926b8409480a8452b3a121f58"
I20251212 03:12:27.143896 21178 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 0 FOLLOWER]: Advancing to term 1
I20251212 03:12:27.143872 21102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95ad0197220142e8ae9cb7546f4a2bcd"
I20251212 03:12:27.143958 21102 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 0 FOLLOWER]: Advancing to term 1
I20251212 03:12:27.144459 21178 raft_consensus.cc:2468] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ac58b6f4a0b43eaad688ad29d00523c in term 1.
I20251212 03:12:27.144639 21102 raft_consensus.cc:2468] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ac58b6f4a0b43eaad688ad29d00523c in term 1.
I20251212 03:12:27.144640 21217 leader_election.cc:304] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80f56db926b8409480a8452b3a121f58, 8ac58b6f4a0b43eaad688ad29d00523c; no voters:
I20251212 03:12:27.144748 21305 raft_consensus.cc:2804] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 1 FOLLOWER]: Leader election won for term 1
I20251212 03:12:27.144865 21305 raft_consensus.cc:697] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 1 LEADER]: Becoming Leader. State: Replica: 8ac58b6f4a0b43eaad688ad29d00523c, State: Running, Role: LEADER
I20251212 03:12:27.144928 21305 consensus_queue.cc:237] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:27.145460 20999 catalog_manager.cc:5654] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ac58b6f4a0b43eaad688ad29d00523c (127.19.253.195). New cstate: current_term: 1 leader_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:27.152489 21178 raft_consensus.cc:1275] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 1 FOLLOWER]: Refusing update from remote peer 8ac58b6f4a0b43eaad688ad29d00523c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251212 03:12:27.152504 21102 raft_consensus.cc:1275] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 1 FOLLOWER]: Refusing update from remote peer 8ac58b6f4a0b43eaad688ad29d00523c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251212 03:12:27.152714 21305 consensus_queue.cc:1048] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [LEADER]: Connected to new peer: Peer: permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251212 03:12:27.152835 21310 consensus_queue.cc:1048] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [LEADER]: Connected to new peer: Peer: permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251212 03:12:27.153965 21317 mvcc.cc:204] Tried to move back new op lower bound from 7231525466735362048 to 7231525466706157568. Current Snapshot: MvccSnapshot[applied={T|T < 7231525466735362048}]
I20251212 03:12:27.453706 21304 raft_consensus.cc:493] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8ac58b6f4a0b43eaad688ad29d00523c)
I20251212 03:12:27.453814 21304 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 1 FOLLOWER]: Advancing to term 2
I20251212 03:12:27.454581 21304 raft_consensus.cc:515] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:27.454806 21304 leader_election.cc:290] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [CANDIDATE]: Term 2 election: Requested vote from peers 80f56db926b8409480a8452b3a121f58 (127.19.253.194:33281), 8ac58b6f4a0b43eaad688ad29d00523c (127.19.253.195:40203)
I20251212 03:12:27.457559 21253 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "8ac58b6f4a0b43eaad688ad29d00523c"
I20251212 03:12:27.457713 21178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "80f56db926b8409480a8452b3a121f58"
I20251212 03:12:27.457923 21066 leader_election.cc:304] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95ad0197220142e8ae9cb7546f4a2bcd; no voters: 80f56db926b8409480a8452b3a121f58, 8ac58b6f4a0b43eaad688ad29d00523c
I20251212 03:12:27.458039 21304 raft_consensus.cc:2749] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251212 03:12:27.465317 21102 raft_consensus.cc:1240] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 2 FOLLOWER]: Rejecting Update request from peer 8ac58b6f4a0b43eaad688ad29d00523c for earlier term 1. Current term is 2. Ops: []
I20251212 03:12:27.465615 21334 consensus_queue.cc:1059] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 }, Status: INVALID_TERM, Last received: 1.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20251212 03:12:27.469834 21334 raft_consensus.cc:3055] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 1 LEADER]: Stepping down as leader of term 1
I20251212 03:12:27.469895 21334 raft_consensus.cc:740] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8ac58b6f4a0b43eaad688ad29d00523c, State: Running, Role: LEADER
I20251212 03:12:27.469957 21334 consensus_queue.cc:260] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 17, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:27.470054 21334 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 1 FOLLOWER]: Advancing to term 2
I20251212 03:12:27.523166 21308 raft_consensus.cc:493] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8ac58b6f4a0b43eaad688ad29d00523c)
I20251212 03:12:27.523274 21308 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 1 FOLLOWER]: Advancing to term 2
I20251212 03:12:27.524008 21308 raft_consensus.cc:515] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:27.524211 21308 leader_election.cc:290] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [CANDIDATE]: Term 2 election: Requested vote from peers 95ad0197220142e8ae9cb7546f4a2bcd (127.19.253.193:36423), 8ac58b6f4a0b43eaad688ad29d00523c (127.19.253.195:40203)
I20251212 03:12:27.526669 21102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "80f56db926b8409480a8452b3a121f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "95ad0197220142e8ae9cb7546f4a2bcd"
I20251212 03:12:27.526762 21102 raft_consensus.cc:2393] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 80f56db926b8409480a8452b3a121f58 in current term 2: Already voted for candidate 95ad0197220142e8ae9cb7546f4a2bcd in this term.
I20251212 03:12:27.526973 21253 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "80f56db926b8409480a8452b3a121f58" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "8ac58b6f4a0b43eaad688ad29d00523c"
I20251212 03:12:27.527667 21253 raft_consensus.cc:2468] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80f56db926b8409480a8452b3a121f58 in term 2.
I20251212 03:12:27.527894 21144 leader_election.cc:304] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80f56db926b8409480a8452b3a121f58, 8ac58b6f4a0b43eaad688ad29d00523c; no voters: 95ad0197220142e8ae9cb7546f4a2bcd
I20251212 03:12:27.528049 21308 raft_consensus.cc:2804] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 2 FOLLOWER]: Leader election won for term 2
I20251212 03:12:27.528180 21308 raft_consensus.cc:697] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 2 LEADER]: Becoming Leader. State: Replica: 80f56db926b8409480a8452b3a121f58, State: Running, Role: LEADER
I20251212 03:12:27.528245 21308 consensus_queue.cc:237] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 17, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:27.528739 20998 catalog_manager.cc:5654] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 reported cstate change: term changed from 1 to 2, leader changed from 8ac58b6f4a0b43eaad688ad29d00523c (127.19.253.195) to 80f56db926b8409480a8452b3a121f58 (127.19.253.194). New cstate: current_term: 2 leader_uuid: "80f56db926b8409480a8452b3a121f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:27.564527 21304 raft_consensus.cc:493] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:27.564627 21304 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 2 FOLLOWER]: Advancing to term 3
I20251212 03:12:27.565328 21304 raft_consensus.cc:515] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:27.565505 21304 leader_election.cc:290] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [CANDIDATE]: Term 3 election: Requested vote from peers 80f56db926b8409480a8452b3a121f58 (127.19.253.194:33281), 8ac58b6f4a0b43eaad688ad29d00523c (127.19.253.195:40203)
I20251212 03:12:27.565739 21253 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" candidate_term: 3 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "8ac58b6f4a0b43eaad688ad29d00523c"
I20251212 03:12:27.565819 21253 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 2 FOLLOWER]: Advancing to term 3
I20251212 03:12:27.565740 21178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" candidate_term: 3 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "80f56db926b8409480a8452b3a121f58"
I20251212 03:12:27.566438 21253 raft_consensus.cc:2468] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95ad0197220142e8ae9cb7546f4a2bcd in term 3.
I20251212 03:12:27.566632 21068 leader_election.cc:304] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8ac58b6f4a0b43eaad688ad29d00523c, 95ad0197220142e8ae9cb7546f4a2bcd; no voters: 80f56db926b8409480a8452b3a121f58
I20251212 03:12:27.566740 21304 raft_consensus.cc:2804] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 3 FOLLOWER]: Leader election won for term 3
I20251212 03:12:27.566871 21304 raft_consensus.cc:697] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 3 LEADER]: Becoming Leader. State: Replica: 95ad0197220142e8ae9cb7546f4a2bcd, State: Running, Role: LEADER
I20251212 03:12:27.566937 21304 consensus_queue.cc:237] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 17, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:27.620630 20997 catalog_manager.cc:5654] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd reported cstate change: term changed from 2 to 3, leader changed from 80f56db926b8409480a8452b3a121f58 (127.19.253.194) to 95ad0197220142e8ae9cb7546f4a2bcd (127.19.253.193). New cstate: current_term: 3 leader_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:27.636269 21102 raft_consensus.cc:1240] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 3 LEADER]: Rejecting Update request from peer 80f56db926b8409480a8452b3a121f58 for earlier term 2. Current term is 3. Ops: []
I20251212 03:12:27.636584 21308 consensus_queue.cc:1059] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20251212 03:12:27.636682 21308 consensus_queue.cc:1243] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [LEADER]: Peer 95ad0197220142e8ae9cb7546f4a2bcd log is divergent from this leader: its last log entry 3.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20251212 03:12:27.637790 21337 raft_consensus.cc:3055] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 2 LEADER]: Stepping down as leader of term 2
I20251212 03:12:27.637830 21337 raft_consensus.cc:740] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 80f56db926b8409480a8452b3a121f58, State: Running, Role: LEADER
I20251212 03:12:27.637887 21337 consensus_queue.cc:260] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 2.18, Last appended by leader: 18, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:27.637967 21337 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 2 FOLLOWER]: Advancing to term 3
I20251212 03:12:27.661229 21178 raft_consensus.cc:1275] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 3 FOLLOWER]: Refusing update from remote peer 95ad0197220142e8ae9cb7546f4a2bcd: Log matching property violated. Preceding OpId in replica: term: 2 index: 18. Preceding OpId from leader: term: 3 index: 18. (term mismatch)
I20251212 03:12:27.661301 21178 pending_rounds.cc:85] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58: Aborting all ops after (but not including) 17
I20251212 03:12:27.661315 21178 pending_rounds.cc:107] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58: Aborting uncommitted NO_OP operation due to leader change: 2.18
I20251212 03:12:27.661330 21178 raft_consensus.cc:2889] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:27.661549 21304 consensus_queue.cc:1048] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20251212 03:12:27.666296 21253 raft_consensus.cc:1275] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 3 FOLLOWER]: Refusing update from remote peer 95ad0197220142e8ae9cb7546f4a2bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 3 index: 18. (index mismatch)
I20251212 03:12:27.666514 21304 consensus_queue.cc:1048] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20251212 03:12:27.767763 21334 raft_consensus.cc:493] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 95ad0197220142e8ae9cb7546f4a2bcd)
I20251212 03:12:27.767854 21334 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 3 FOLLOWER]: Advancing to term 4
I20251212 03:12:27.768520 21334 raft_consensus.cc:515] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:27.768666 21334 leader_election.cc:290] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [CANDIDATE]: Term 4 election: Requested vote from peers 95ad0197220142e8ae9cb7546f4a2bcd (127.19.253.193:36423), 80f56db926b8409480a8452b3a121f58 (127.19.253.194:33281)
I20251212 03:12:27.768860 21102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "95ad0197220142e8ae9cb7546f4a2bcd"
I20251212 03:12:27.768874 21178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "80f56db926b8409480a8452b3a121f58"
I20251212 03:12:27.769091 21219 leader_election.cc:304] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac58b6f4a0b43eaad688ad29d00523c; no voters: 80f56db926b8409480a8452b3a121f58, 95ad0197220142e8ae9cb7546f4a2bcd
I20251212 03:12:27.769199 21334 raft_consensus.cc:2749] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251212 03:12:27.781160 21253 raft_consensus.cc:1240] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 4 FOLLOWER]: Rejecting Update request from peer 95ad0197220142e8ae9cb7546f4a2bcd for earlier term 3. Current term is 4. Ops: []
I20251212 03:12:27.781441 21342 consensus_queue.cc:1059] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 }, Status: INVALID_TERM, Last received: 3.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20251212 03:12:27.810352 21338 raft_consensus.cc:3055] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 3 LEADER]: Stepping down as leader of term 3
I20251212 03:12:27.810465 21338 raft_consensus.cc:740] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 95ad0197220142e8ae9cb7546f4a2bcd, State: Running, Role: LEADER
I20251212 03:12:27.810541 21338 consensus_queue.cc:260] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 3.18, Last appended by leader: 18, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:27.810658 21338 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 3 FOLLOWER]: Advancing to term 4
I20251212 03:12:27.863402 21337 raft_consensus.cc:493] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 95ad0197220142e8ae9cb7546f4a2bcd)
I20251212 03:12:27.863525 21337 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 3 FOLLOWER]: Advancing to term 4
I20251212 03:12:27.864199 21337 raft_consensus.cc:515] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:27.864349 21337 leader_election.cc:290] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [CANDIDATE]: Term 4 election: Requested vote from peers 95ad0197220142e8ae9cb7546f4a2bcd (127.19.253.193:36423), 8ac58b6f4a0b43eaad688ad29d00523c (127.19.253.195:40203)
I20251212 03:12:27.864676 21102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "80f56db926b8409480a8452b3a121f58" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "95ad0197220142e8ae9cb7546f4a2bcd"
I20251212 03:12:27.864676 21253 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "80f56db926b8409480a8452b3a121f58" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "8ac58b6f4a0b43eaad688ad29d00523c"
I20251212 03:12:27.864799 21253 raft_consensus.cc:2393] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 80f56db926b8409480a8452b3a121f58 in current term 4: Already voted for candidate 8ac58b6f4a0b43eaad688ad29d00523c in this term.
I20251212 03:12:27.865365 21102 raft_consensus.cc:2468] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80f56db926b8409480a8452b3a121f58 in term 4.
I20251212 03:12:27.865592 21144 leader_election.cc:304] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80f56db926b8409480a8452b3a121f58, 95ad0197220142e8ae9cb7546f4a2bcd; no voters: 8ac58b6f4a0b43eaad688ad29d00523c
I20251212 03:12:27.865680 21337 raft_consensus.cc:2804] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 4 FOLLOWER]: Leader election won for term 4
I20251212 03:12:27.865732 21337 raft_consensus.cc:697] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 4 LEADER]: Becoming Leader. State: Replica: 80f56db926b8409480a8452b3a121f58, State: Running, Role: LEADER
I20251212 03:12:27.865789 21337 consensus_queue.cc:237] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 3.18, Last appended by leader: 18, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:27.866395 20997 catalog_manager.cc:5654] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 reported cstate change: term changed from 3 to 4, leader changed from 95ad0197220142e8ae9cb7546f4a2bcd (127.19.253.193) to 80f56db926b8409480a8452b3a121f58 (127.19.253.194). New cstate: current_term: 4 leader_uuid: "80f56db926b8409480a8452b3a121f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:27.891685 21334 raft_consensus.cc:493] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:27.891774 21334 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 4 FOLLOWER]: Advancing to term 5
I20251212 03:12:27.892387 21334 raft_consensus.cc:515] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:27.892527 21334 leader_election.cc:290] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [CANDIDATE]: Term 5 election: Requested vote from peers 95ad0197220142e8ae9cb7546f4a2bcd (127.19.253.193:36423), 80f56db926b8409480a8452b3a121f58 (127.19.253.194:33281)
I20251212 03:12:27.892760 21102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" candidate_term: 5 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "95ad0197220142e8ae9cb7546f4a2bcd"
I20251212 03:12:27.892853 21102 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 4 FOLLOWER]: Advancing to term 5
I20251212 03:12:27.892764 21178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" candidate_term: 5 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "80f56db926b8409480a8452b3a121f58"
I20251212 03:12:27.893445 21102 raft_consensus.cc:2468] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ac58b6f4a0b43eaad688ad29d00523c in term 5.
I20251212 03:12:27.893612 21219 leader_election.cc:304] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8ac58b6f4a0b43eaad688ad29d00523c, 95ad0197220142e8ae9cb7546f4a2bcd; no voters: 80f56db926b8409480a8452b3a121f58
I20251212 03:12:27.893710 21334 raft_consensus.cc:2804] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 5 FOLLOWER]: Leader election won for term 5
I20251212 03:12:27.893764 21334 raft_consensus.cc:697] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 5 LEADER]: Becoming Leader. State: Replica: 8ac58b6f4a0b43eaad688ad29d00523c, State: Running, Role: LEADER
I20251212 03:12:27.893818 21334 consensus_queue.cc:237] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 3.18, Last appended by leader: 18, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:27.894316 20997 catalog_manager.cc:5654] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c reported cstate change: term changed from 4 to 5, leader changed from 80f56db926b8409480a8452b3a121f58 (127.19.253.194) to 8ac58b6f4a0b43eaad688ad29d00523c (127.19.253.195). New cstate: current_term: 5 leader_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:27.963694 21102 raft_consensus.cc:1240] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 5 FOLLOWER]: Rejecting Update request from peer 80f56db926b8409480a8452b3a121f58 for earlier term 4. Current term is 5. Ops: []
I20251212 03:12:27.964015 21337 consensus_queue.cc:1059] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20251212 03:12:27.978852 21253 raft_consensus.cc:1240] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 5 LEADER]: Rejecting Update request from peer 80f56db926b8409480a8452b3a121f58 for earlier term 4. Current term is 5. Ops: []
I20251212 03:12:27.979113 21308 consensus_queue.cc:1059] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20251212 03:12:27.979243 21308 consensus_queue.cc:1243] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [LEADER]: Peer 8ac58b6f4a0b43eaad688ad29d00523c log is divergent from this leader: its last log entry 5.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20251212 03:12:27.991230 21337 raft_consensus.cc:3055] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 4 LEADER]: Stepping down as leader of term 4
I20251212 03:12:27.991312 21337 raft_consensus.cc:740] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 80f56db926b8409480a8452b3a121f58, State: Running, Role: LEADER
I20251212 03:12:27.991375 21337 consensus_queue.cc:260] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 4.19, Last appended by leader: 19, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:27.991480 21337 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 4 FOLLOWER]: Advancing to term 5
I20251212 03:12:28.017767 21102 raft_consensus.cc:1275] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 5 FOLLOWER]: Refusing update from remote peer 8ac58b6f4a0b43eaad688ad29d00523c: Log matching property violated. Preceding OpId in replica: term: 3 index: 18. Preceding OpId from leader: term: 5 index: 19. (index mismatch)
I20251212 03:12:28.017767 21178 raft_consensus.cc:1275] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 5 FOLLOWER]: Refusing update from remote peer 8ac58b6f4a0b43eaad688ad29d00523c: Log matching property violated. Preceding OpId in replica: term: 4 index: 19. Preceding OpId from leader: term: 5 index: 19. (term mismatch)
I20251212 03:12:28.017901 21178 pending_rounds.cc:85] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58: Aborting all ops after (but not including) 18
I20251212 03:12:28.017922 21178 pending_rounds.cc:107] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58: Aborting uncommitted NO_OP operation due to leader change: 4.19
I20251212 03:12:28.017949 21178 raft_consensus.cc:2889] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:28.018100 21310 consensus_queue.cc:1048] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [LEADER]: Connected to new peer: Peer: permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20251212 03:12:28.018249 21310 consensus_queue.cc:1048] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [LEADER]: Connected to new peer: Peer: permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20251212 03:12:28.119933 21338 raft_consensus.cc:493] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8ac58b6f4a0b43eaad688ad29d00523c)
I20251212 03:12:28.120028 21338 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 5 FOLLOWER]: Advancing to term 6
I20251212 03:12:28.120714 21338 raft_consensus.cc:515] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:28.120867 21338 leader_election.cc:290] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [CANDIDATE]: Term 6 election: Requested vote from peers 80f56db926b8409480a8452b3a121f58 (127.19.253.194:33281), 8ac58b6f4a0b43eaad688ad29d00523c (127.19.253.195:40203)
I20251212 03:12:28.121043 21178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" candidate_term: 6 candidate_status { last_received { term: 5 index: 19 } } ignore_live_leader: false dest_uuid: "80f56db926b8409480a8452b3a121f58"
I20251212 03:12:28.121063 21253 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" candidate_term: 6 candidate_status { last_received { term: 5 index: 19 } } ignore_live_leader: false dest_uuid: "8ac58b6f4a0b43eaad688ad29d00523c"
I20251212 03:12:28.121274 21068 leader_election.cc:304] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95ad0197220142e8ae9cb7546f4a2bcd; no voters: 80f56db926b8409480a8452b3a121f58, 8ac58b6f4a0b43eaad688ad29d00523c
I20251212 03:12:28.121367 21338 raft_consensus.cc:2749] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251212 03:12:28.127875 21102 raft_consensus.cc:1240] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 6 FOLLOWER]: Rejecting Update request from peer 8ac58b6f4a0b43eaad688ad29d00523c for earlier term 5. Current term is 6. Ops: []
I20251212 03:12:28.128111 21334 consensus_queue.cc:1059] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 }, Status: INVALID_TERM, Last received: 5.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20251212 03:12:28.227341 21102 raft_consensus.cc:1240] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 6 FOLLOWER]: Rejecting Update request from peer 8ac58b6f4a0b43eaad688ad29d00523c for earlier term 5. Current term is 6. Ops: []
I20251212 03:12:28.227663 21310 consensus_queue.cc:1059] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 }, Status: INVALID_TERM, Last received: 5.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20251212 03:12:28.227809 21338 raft_consensus.cc:493] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:28.227882 21338 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 6 FOLLOWER]: Advancing to term 7
I20251212 03:12:28.228519 21338 raft_consensus.cc:515] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:28.228648 21338 leader_election.cc:290] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [CANDIDATE]: Term 7 election: Requested vote from peers 80f56db926b8409480a8452b3a121f58 (127.19.253.194:33281), 8ac58b6f4a0b43eaad688ad29d00523c (127.19.253.195:40203)
I20251212 03:12:28.228837 21253 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" candidate_term: 7 candidate_status { last_received { term: 5 index: 19 } } ignore_live_leader: false dest_uuid: "8ac58b6f4a0b43eaad688ad29d00523c"
I20251212 03:12:28.228834 21178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" candidate_term: 7 candidate_status { last_received { term: 5 index: 19 } } ignore_live_leader: false dest_uuid: "80f56db926b8409480a8452b3a121f58"
I20251212 03:12:28.229159 21066 leader_election.cc:304] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95ad0197220142e8ae9cb7546f4a2bcd; no voters: 80f56db926b8409480a8452b3a121f58, 8ac58b6f4a0b43eaad688ad29d00523c
I20251212 03:12:28.229285 21338 raft_consensus.cc:2749] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251212 03:12:28.252664 21343 raft_consensus.cc:3055] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 5 LEADER]: Stepping down as leader of term 5
I20251212 03:12:28.252756 21343 raft_consensus.cc:740] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 8ac58b6f4a0b43eaad688ad29d00523c, State: Running, Role: LEADER
I20251212 03:12:28.252838 21343 consensus_queue.cc:260] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 5.19, Last appended by leader: 19, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:28.252992 21343 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 5 FOLLOWER]: Advancing to term 6
I20251212 03:12:28.316344 21337 raft_consensus.cc:493] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 8ac58b6f4a0b43eaad688ad29d00523c)
I20251212 03:12:28.316457 21337 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 5 FOLLOWER]: Advancing to term 6
I20251212 03:12:28.317114 21337 raft_consensus.cc:515] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:28.317305 21337 leader_election.cc:290] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [CANDIDATE]: Term 6 election: Requested vote from peers 95ad0197220142e8ae9cb7546f4a2bcd (127.19.253.193:36423), 8ac58b6f4a0b43eaad688ad29d00523c (127.19.253.195:40203)
I20251212 03:12:28.317577 21102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "80f56db926b8409480a8452b3a121f58" candidate_term: 6 candidate_status { last_received { term: 5 index: 19 } } ignore_live_leader: false dest_uuid: "95ad0197220142e8ae9cb7546f4a2bcd"
I20251212 03:12:28.317577 21253 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "80f56db926b8409480a8452b3a121f58" candidate_term: 6 candidate_status { last_received { term: 5 index: 19 } } ignore_live_leader: false dest_uuid: "8ac58b6f4a0b43eaad688ad29d00523c"
I20251212 03:12:28.317688 21102 raft_consensus.cc:2368] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 80f56db926b8409480a8452b3a121f58 for earlier term 6. Current term is 7.
I20251212 03:12:28.317890 21144 leader_election.cc:400] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [CANDIDATE]: Term 6 election: Vote denied by peer 95ad0197220142e8ae9cb7546f4a2bcd (127.19.253.193:36423) with higher term. Message: Invalid argument: T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 80f56db926b8409480a8452b3a121f58 for earlier term 6. Current term is 7.
I20251212 03:12:28.317953 21144 leader_election.cc:403] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20251212 03:12:28.318022 21337 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 6 FOLLOWER]: Advancing to term 7
I20251212 03:12:28.318274 21253 raft_consensus.cc:2468] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80f56db926b8409480a8452b3a121f58 in term 6.
I20251212 03:12:28.318673 21337 raft_consensus.cc:2749] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 95ad0197220142e8ae9cb7546f4a2bcd (127.19.253.193:36423) with higher term. Message: Invalid argument: T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 80f56db926b8409480a8452b3a121f58 for earlier term 6. Current term is 7.
I20251212 03:12:28.453063 21338 raft_consensus.cc:493] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:28.453164 21338 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 7 FOLLOWER]: Advancing to term 8
I20251212 03:12:28.453948 21338 raft_consensus.cc:515] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:28.454129 21338 leader_election.cc:290] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [CANDIDATE]: Term 8 election: Requested vote from peers 80f56db926b8409480a8452b3a121f58 (127.19.253.194:33281), 8ac58b6f4a0b43eaad688ad29d00523c (127.19.253.195:40203)
I20251212 03:12:28.454393 21253 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" candidate_term: 8 candidate_status { last_received { term: 5 index: 19 } } ignore_live_leader: false dest_uuid: "8ac58b6f4a0b43eaad688ad29d00523c"
I20251212 03:12:28.454478 21253 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 6 FOLLOWER]: Advancing to term 8
I20251212 03:12:28.454406 21178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" candidate_term: 8 candidate_status { last_received { term: 5 index: 19 } } ignore_live_leader: false dest_uuid: "80f56db926b8409480a8452b3a121f58"
I20251212 03:12:28.454516 21178 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 7 FOLLOWER]: Advancing to term 8
I20251212 03:12:28.455030 21253 raft_consensus.cc:2468] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95ad0197220142e8ae9cb7546f4a2bcd in term 8.
I20251212 03:12:28.455085 21178 raft_consensus.cc:2468] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95ad0197220142e8ae9cb7546f4a2bcd in term 8.
I20251212 03:12:28.455214 21068 leader_election.cc:304] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac58b6f4a0b43eaad688ad29d00523c, 95ad0197220142e8ae9cb7546f4a2bcd; no voters:
I20251212 03:12:28.455309 21338 raft_consensus.cc:2804] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 8 FOLLOWER]: Leader election won for term 8
I20251212 03:12:28.455410 21338 raft_consensus.cc:697] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 8 LEADER]: Becoming Leader. State: Replica: 95ad0197220142e8ae9cb7546f4a2bcd, State: Running, Role: LEADER
I20251212 03:12:28.455474 21338 consensus_queue.cc:237] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 5.19, Last appended by leader: 19, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:28.455969 20997 catalog_manager.cc:5654] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd reported cstate change: term changed from 5 to 8, leader changed from 8ac58b6f4a0b43eaad688ad29d00523c (127.19.253.195) to 95ad0197220142e8ae9cb7546f4a2bcd (127.19.253.193). New cstate: current_term: 8 leader_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:28.560518 21178 raft_consensus.cc:1275] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 8 FOLLOWER]: Refusing update from remote peer 95ad0197220142e8ae9cb7546f4a2bcd: Log matching property violated. Preceding OpId in replica: term: 5 index: 19. Preceding OpId from leader: term: 8 index: 20. (index mismatch)
I20251212 03:12:28.560814 21338 consensus_queue.cc:1048] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20251212 03:12:28.569044 21343 raft_consensus.cc:493] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:28.569131 21343 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 8 FOLLOWER]: Advancing to term 9
I20251212 03:12:28.569801 21343 raft_consensus.cc:515] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:28.569983 21343 leader_election.cc:290] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [CANDIDATE]: Term 9 election: Requested vote from peers 95ad0197220142e8ae9cb7546f4a2bcd (127.19.253.193:36423), 80f56db926b8409480a8452b3a121f58 (127.19.253.194:33281)
I20251212 03:12:28.570161 21102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" candidate_term: 9 candidate_status { last_received { term: 5 index: 19 } } ignore_live_leader: false dest_uuid: "95ad0197220142e8ae9cb7546f4a2bcd"
I20251212 03:12:28.570248 21178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" candidate_term: 9 candidate_status { last_received { term: 5 index: 19 } } ignore_live_leader: false dest_uuid: "80f56db926b8409480a8452b3a121f58"
I20251212 03:12:28.570451 21219 leader_election.cc:304] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac58b6f4a0b43eaad688ad29d00523c; no voters: 80f56db926b8409480a8452b3a121f58, 95ad0197220142e8ae9cb7546f4a2bcd
I20251212 03:12:28.570546 21343 raft_consensus.cc:2749] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251212 03:12:28.574875 21253 raft_consensus.cc:1240] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 9 FOLLOWER]: Rejecting Update request from peer 95ad0197220142e8ae9cb7546f4a2bcd for earlier term 8. Current term is 9. Ops: []
I20251212 03:12:28.575249 21338 consensus_queue.cc:1059] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20251212 03:12:28.636377 21253 raft_consensus.cc:1240] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 9 FOLLOWER]: Rejecting Update request from peer 95ad0197220142e8ae9cb7546f4a2bcd for earlier term 8. Current term is 9. Ops: [8.20-8.20]
I20251212 03:12:28.636675 21338 consensus_queue.cc:1059] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 }, Status: INVALID_TERM, Last received: 5.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20251212 03:12:28.662756 21337 raft_consensus.cc:493] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 95ad0197220142e8ae9cb7546f4a2bcd)
I20251212 03:12:28.662878 21337 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 8 FOLLOWER]: Advancing to term 9
I20251212 03:12:28.663602 21337 raft_consensus.cc:515] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:28.663769 21337 leader_election.cc:290] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [CANDIDATE]: Term 9 election: Requested vote from peers 95ad0197220142e8ae9cb7546f4a2bcd (127.19.253.193:36423), 8ac58b6f4a0b43eaad688ad29d00523c (127.19.253.195:40203)
I20251212 03:12:28.664048 21102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "80f56db926b8409480a8452b3a121f58" candidate_term: 9 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "95ad0197220142e8ae9cb7546f4a2bcd"
I20251212 03:12:28.664048 21253 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "80f56db926b8409480a8452b3a121f58" candidate_term: 9 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "8ac58b6f4a0b43eaad688ad29d00523c"
I20251212 03:12:28.664234 21253 raft_consensus.cc:2393] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 80f56db926b8409480a8452b3a121f58 in current term 9: Already voted for candidate 8ac58b6f4a0b43eaad688ad29d00523c in this term.
I20251212 03:12:28.664417 21144 leader_election.cc:304] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80f56db926b8409480a8452b3a121f58; no voters: 8ac58b6f4a0b43eaad688ad29d00523c, 95ad0197220142e8ae9cb7546f4a2bcd
I20251212 03:12:28.664518 21337 raft_consensus.cc:2749] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251212 03:12:28.684937 21178 raft_consensus.cc:1240] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 9 FOLLOWER]: Rejecting Update request from peer 95ad0197220142e8ae9cb7546f4a2bcd for earlier term 8. Current term is 9. Ops: []
I20251212 03:12:28.685284 21344 consensus_queue.cc:1059] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 }, Status: INVALID_TERM, Last received: 8.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20251212 03:12:28.718950 21343 raft_consensus.cc:493] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:28.719058 21343 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 9 FOLLOWER]: Advancing to term 10
I20251212 03:12:28.719739 21343 raft_consensus.cc:515] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:28.719902 21343 leader_election.cc:290] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [CANDIDATE]: Term 10 election: Requested vote from peers 95ad0197220142e8ae9cb7546f4a2bcd (127.19.253.193:36423), 80f56db926b8409480a8452b3a121f58 (127.19.253.194:33281)
I20251212 03:12:28.720155 21178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" candidate_term: 10 candidate_status { last_received { term: 5 index: 19 } } ignore_live_leader: false dest_uuid: "80f56db926b8409480a8452b3a121f58"
I20251212 03:12:28.720163 21102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" candidate_term: 10 candidate_status { last_received { term: 5 index: 19 } } ignore_live_leader: false dest_uuid: "95ad0197220142e8ae9cb7546f4a2bcd"
I20251212 03:12:28.720280 21178 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 9 FOLLOWER]: Advancing to term 10
I20251212 03:12:28.720888 21178 raft_consensus.cc:2410] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 8ac58b6f4a0b43eaad688ad29d00523c for term 10 because replica has last-logged OpId of term: 8 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 19.
I20251212 03:12:28.721114 21217 leader_election.cc:304] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac58b6f4a0b43eaad688ad29d00523c; no voters: 80f56db926b8409480a8452b3a121f58, 95ad0197220142e8ae9cb7546f4a2bcd
I20251212 03:12:28.721241 21343 raft_consensus.cc:2749] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20251212 03:12:28.723873 21253 raft_consensus.cc:1240] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 10 FOLLOWER]: Rejecting Update request from peer 95ad0197220142e8ae9cb7546f4a2bcd for earlier term 8. Current term is 10. Ops: [8.20-8.20]
I20251212 03:12:28.724218 21338 consensus_queue.cc:1059] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 }, Status: INVALID_TERM, Last received: 5.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20251212 03:12:28.750384 21349 raft_consensus.cc:3055] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 8 LEADER]: Stepping down as leader of term 8
I20251212 03:12:28.750481 21349 raft_consensus.cc:740] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 95ad0197220142e8ae9cb7546f4a2bcd, State: Running, Role: LEADER
I20251212 03:12:28.750557 21349 consensus_queue.cc:260] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 8.20, Last appended by leader: 20, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:28.750670 21349 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 8 FOLLOWER]: Advancing to term 9
I20251212 03:12:28.804118 21337 raft_consensus.cc:493] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:28.804244 21337 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 10 FOLLOWER]: Advancing to term 11
I20251212 03:12:28.804912 21337 raft_consensus.cc:515] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:28.805084 21337 leader_election.cc:290] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [CANDIDATE]: Term 11 election: Requested vote from peers 95ad0197220142e8ae9cb7546f4a2bcd (127.19.253.193:36423), 8ac58b6f4a0b43eaad688ad29d00523c (127.19.253.195:40203)
I20251212 03:12:28.805346 21102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "80f56db926b8409480a8452b3a121f58" candidate_term: 11 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "95ad0197220142e8ae9cb7546f4a2bcd"
I20251212 03:12:28.805359 21253 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "80f56db926b8409480a8452b3a121f58" candidate_term: 11 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "8ac58b6f4a0b43eaad688ad29d00523c"
I20251212 03:12:28.805446 21102 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 9 FOLLOWER]: Advancing to term 11
I20251212 03:12:28.805459 21253 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 10 FOLLOWER]: Advancing to term 11
I20251212 03:12:28.805966 21253 raft_consensus.cc:2468] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80f56db926b8409480a8452b3a121f58 in term 11.
I20251212 03:12:28.806033 21102 raft_consensus.cc:2468] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80f56db926b8409480a8452b3a121f58 in term 11.
I20251212 03:12:28.806165 21144 leader_election.cc:304] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80f56db926b8409480a8452b3a121f58, 8ac58b6f4a0b43eaad688ad29d00523c; no voters:
I20251212 03:12:28.806299 21337 raft_consensus.cc:2804] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 11 FOLLOWER]: Leader election won for term 11
I20251212 03:12:28.806404 21337 raft_consensus.cc:697] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 11 LEADER]: Becoming Leader. State: Replica: 80f56db926b8409480a8452b3a121f58, State: Running, Role: LEADER
I20251212 03:12:28.806447 21337 consensus_queue.cc:237] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 8.20, Last appended by leader: 20, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:28.806937 20997 catalog_manager.cc:5654] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 reported cstate change: term changed from 8 to 11, leader changed from 95ad0197220142e8ae9cb7546f4a2bcd (127.19.253.193) to 80f56db926b8409480a8452b3a121f58 (127.19.253.194). New cstate: current_term: 11 leader_uuid: "80f56db926b8409480a8452b3a121f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:28.809129 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:28.810277 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:28.812466 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:28.816640 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:28.824822 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:28.841032 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:28.873270 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:28.914438 21253 raft_consensus.cc:1275] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 11 FOLLOWER]: Refusing update from remote peer 80f56db926b8409480a8452b3a121f58: Log matching property violated. Preceding OpId in replica: term: 5 index: 19. Preceding OpId from leader: term: 11 index: 21. (index mismatch)
I20251212 03:12:28.914733 21337 consensus_queue.cc:1048] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s
I20251212 03:12:28.916093 21102 raft_consensus.cc:1275] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 11 FOLLOWER]: Refusing update from remote peer 80f56db926b8409480a8452b3a121f58: Log matching property violated. Preceding OpId in replica: term: 8 index: 20. Preceding OpId from leader: term: 11 index: 21. (index mismatch)
I20251212 03:12:28.916298 21352 consensus_queue.cc:1048] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20251212 03:12:28.937538 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:29.017282 21349 raft_consensus.cc:493] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 11 FOLLOWER]: Starting leader election (detected failure of leader 80f56db926b8409480a8452b3a121f58)
I20251212 03:12:29.017390 21349 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 11 FOLLOWER]: Advancing to term 12
I20251212 03:12:29.018095 21349 raft_consensus.cc:515] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:29.018309 21349 leader_election.cc:290] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [CANDIDATE]: Term 12 election: Requested vote from peers 80f56db926b8409480a8452b3a121f58 (127.19.253.194:33281), 8ac58b6f4a0b43eaad688ad29d00523c (127.19.253.195:40203)
I20251212 03:12:29.018574 21253 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" candidate_term: 12 candidate_status { last_received { term: 11 index: 21 } } ignore_live_leader: false dest_uuid: "8ac58b6f4a0b43eaad688ad29d00523c"
I20251212 03:12:29.018579 21178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" candidate_term: 12 candidate_status { last_received { term: 11 index: 21 } } ignore_live_leader: false dest_uuid: "80f56db926b8409480a8452b3a121f58"
I20251212 03:12:29.018826 21066 leader_election.cc:304] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95ad0197220142e8ae9cb7546f4a2bcd; no voters: 80f56db926b8409480a8452b3a121f58, 8ac58b6f4a0b43eaad688ad29d00523c
I20251212 03:12:29.018921 21349 raft_consensus.cc:2749] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20251212 03:12:29.033236 21102 raft_consensus.cc:1240] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 12 FOLLOWER]: Rejecting Update request from peer 80f56db926b8409480a8452b3a121f58 for earlier term 11. Current term is 12. Ops: []
I20251212 03:12:29.033545 21350 consensus_queue.cc:1059] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 }, Status: INVALID_TERM, Last received: 11.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20251212 03:12:29.033684 21350 raft_consensus.cc:3055] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 11 LEADER]: Stepping down as leader of term 11
I20251212 03:12:29.033720 21350 raft_consensus.cc:740] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 80f56db926b8409480a8452b3a121f58, State: Running, Role: LEADER
I20251212 03:12:29.033784 21350 consensus_queue.cc:260] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 11.21, Last appended by leader: 21, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:29.033874 21350 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 11 FOLLOWER]: Advancing to term 12
I20251212 03:12:29.065773 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:29.101135 21343 raft_consensus.cc:493] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 11 FOLLOWER]: Starting leader election (detected failure of leader 80f56db926b8409480a8452b3a121f58)
I20251212 03:12:29.101233 21343 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 11 FOLLOWER]: Advancing to term 12
I20251212 03:12:29.101907 21343 raft_consensus.cc:515] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:29.102072 21343 leader_election.cc:290] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [CANDIDATE]: Term 12 election: Requested vote from peers 95ad0197220142e8ae9cb7546f4a2bcd (127.19.253.193:36423), 80f56db926b8409480a8452b3a121f58 (127.19.253.194:33281)
I20251212 03:12:29.102350 21102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" candidate_term: 12 candidate_status { last_received { term: 11 index: 21 } } ignore_live_leader: false dest_uuid: "95ad0197220142e8ae9cb7546f4a2bcd"
I20251212 03:12:29.102350 21178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" candidate_term: 12 candidate_status { last_received { term: 11 index: 21 } } ignore_live_leader: false dest_uuid: "80f56db926b8409480a8452b3a121f58"
I20251212 03:12:29.102501 21102 raft_consensus.cc:2393] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 8ac58b6f4a0b43eaad688ad29d00523c in current term 12: Already voted for candidate 95ad0197220142e8ae9cb7546f4a2bcd in this term.
I20251212 03:12:29.103065 21178 raft_consensus.cc:2468] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ac58b6f4a0b43eaad688ad29d00523c in term 12.
I20251212 03:12:29.103240 21217 leader_election.cc:304] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80f56db926b8409480a8452b3a121f58, 8ac58b6f4a0b43eaad688ad29d00523c; no voters: 95ad0197220142e8ae9cb7546f4a2bcd
I20251212 03:12:29.103344 21343 raft_consensus.cc:2804] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 12 FOLLOWER]: Leader election won for term 12
I20251212 03:12:29.103452 21343 raft_consensus.cc:697] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 12 LEADER]: Becoming Leader. State: Replica: 8ac58b6f4a0b43eaad688ad29d00523c, State: Running, Role: LEADER
I20251212 03:12:29.103500 21343 consensus_queue.cc:237] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 11.21, Last appended by leader: 21, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:29.103950 20998 catalog_manager.cc:5654] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c reported cstate change: term changed from 11 to 12, leader changed from 80f56db926b8409480a8452b3a121f58 (127.19.253.194) to 8ac58b6f4a0b43eaad688ad29d00523c (127.19.253.195). New cstate: current_term: 12 leader_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:29.164016 21349 raft_consensus.cc:493] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:29.164129 21349 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 12 FOLLOWER]: Advancing to term 13
I20251212 03:12:29.164808 21349 raft_consensus.cc:515] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:29.164963 21349 leader_election.cc:290] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [CANDIDATE]: Term 13 election: Requested vote from peers 80f56db926b8409480a8452b3a121f58 (127.19.253.194:33281), 8ac58b6f4a0b43eaad688ad29d00523c (127.19.253.195:40203)
I20251212 03:12:29.165201 21178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" candidate_term: 13 candidate_status { last_received { term: 11 index: 21 } } ignore_live_leader: false dest_uuid: "80f56db926b8409480a8452b3a121f58"
I20251212 03:12:29.165237 21253 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" candidate_term: 13 candidate_status { last_received { term: 11 index: 21 } } ignore_live_leader: false dest_uuid: "8ac58b6f4a0b43eaad688ad29d00523c"
I20251212 03:12:29.165324 21178 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 12 FOLLOWER]: Advancing to term 13
I20251212 03:12:29.165902 21178 raft_consensus.cc:2468] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95ad0197220142e8ae9cb7546f4a2bcd in term 13.
I20251212 03:12:29.166169 21066 leader_election.cc:304] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80f56db926b8409480a8452b3a121f58, 95ad0197220142e8ae9cb7546f4a2bcd; no voters: 8ac58b6f4a0b43eaad688ad29d00523c
I20251212 03:12:29.166294 21349 raft_consensus.cc:2804] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 13 FOLLOWER]: Leader election won for term 13
I20251212 03:12:29.166357 21349 raft_consensus.cc:697] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 13 LEADER]: Becoming Leader. State: Replica: 95ad0197220142e8ae9cb7546f4a2bcd, State: Running, Role: LEADER
I20251212 03:12:29.166412 21349 consensus_queue.cc:237] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 11.21, Last appended by leader: 21, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:29.166893 20998 catalog_manager.cc:5654] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd reported cstate change: term changed from 12 to 13, leader changed from 8ac58b6f4a0b43eaad688ad29d00523c (127.19.253.195) to 95ad0197220142e8ae9cb7546f4a2bcd (127.19.253.193). New cstate: current_term: 13 leader_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:29.198295 21102 raft_consensus.cc:1240] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 13 LEADER]: Rejecting Update request from peer 8ac58b6f4a0b43eaad688ad29d00523c for earlier term 12. Current term is 13. Ops: []
I20251212 03:12:29.198607 21343 consensus_queue.cc:1059] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20251212 03:12:29.198719 21343 consensus_queue.cc:1243] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [LEADER]: Peer 95ad0197220142e8ae9cb7546f4a2bcd log is divergent from this leader: its last log entry 13.22 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 21
I20251212 03:12:29.198757 21353 raft_consensus.cc:3055] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 12 LEADER]: Stepping down as leader of term 12
I20251212 03:12:29.198815 21353 raft_consensus.cc:740] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 8ac58b6f4a0b43eaad688ad29d00523c, State: Running, Role: LEADER
I20251212 03:12:29.198899 21353 consensus_queue.cc:260] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 12.22, Last appended by leader: 22, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:29.198995 21353 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 12 FOLLOWER]: Advancing to term 13
I20251212 03:12:29.280555 21253 raft_consensus.cc:1275] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 13 FOLLOWER]: Refusing update from remote peer 95ad0197220142e8ae9cb7546f4a2bcd: Log matching property violated. Preceding OpId in replica: term: 12 index: 22. Preceding OpId from leader: term: 13 index: 22. (term mismatch)
I20251212 03:12:29.280632 21253 pending_rounds.cc:85] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c: Aborting all ops after (but not including) 21
I20251212 03:12:29.280645 21253 pending_rounds.cc:107] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c: Aborting uncommitted NO_OP operation due to leader change: 12.22
I20251212 03:12:29.280660 21253 raft_consensus.cc:2889] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251212 03:12:29.280937 21349 consensus_queue.cc:1048] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20251212 03:12:29.282127 21178 raft_consensus.cc:1275] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 13 FOLLOWER]: Refusing update from remote peer 95ad0197220142e8ae9cb7546f4a2bcd: Log matching property violated. Preceding OpId in replica: term: 11 index: 21. Preceding OpId from leader: term: 13 index: 22. (index mismatch)
I20251212 03:12:29.282397 21349 consensus_queue.cc:1048] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20251212 03:12:29.322033 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:29.322255 20471 log_reader.cc:385] T bc80f8d16dd043b187b09967fc93dbe3: removed 1 log segments from log reader
I20251212 03:12:29.322314 20471 log.cc:1079] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd: Deleting log segment in path: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765509114134682-20471-0/minicluster-data/ts-0-root/wals/bc80f8d16dd043b187b09967fc93dbe3/wal-000000001 (ops 1-17)
I20251212 03:12:29.322758 20471 timestamp_advancement-itest.cc:274] Shutting down cluster...
I20251212 03:12:29.322822 20471 master.cc:561] Master@127.19.253.254:39701 shutting down...
I20251212 03:12:29.383212 21350 raft_consensus.cc:493] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 95ad0197220142e8ae9cb7546f4a2bcd)
I20251212 03:12:29.383309 21350 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 13 FOLLOWER]: Advancing to term 14
I20251212 03:12:29.383970 21350 raft_consensus.cc:515] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:29.384125 21350 leader_election.cc:290] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [CANDIDATE]: Term 14 election: Requested vote from peers 95ad0197220142e8ae9cb7546f4a2bcd (127.19.253.193:36423), 8ac58b6f4a0b43eaad688ad29d00523c (127.19.253.195:40203)
I20251212 03:12:29.384320 21102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "80f56db926b8409480a8452b3a121f58" candidate_term: 14 candidate_status { last_received { term: 13 index: 22 } } ignore_live_leader: false dest_uuid: "95ad0197220142e8ae9cb7546f4a2bcd"
I20251212 03:12:29.384357 21253 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "80f56db926b8409480a8452b3a121f58" candidate_term: 14 candidate_status { last_received { term: 13 index: 22 } } ignore_live_leader: false dest_uuid: "8ac58b6f4a0b43eaad688ad29d00523c"
I20251212 03:12:29.384554 21144 leader_election.cc:304] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 80f56db926b8409480a8452b3a121f58; no voters: 8ac58b6f4a0b43eaad688ad29d00523c, 95ad0197220142e8ae9cb7546f4a2bcd
I20251212 03:12:29.384646 21350 raft_consensus.cc:2749] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20251212 03:12:29.385563 21178 raft_consensus.cc:1240] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 14 FOLLOWER]: Rejecting Update request from peer 95ad0197220142e8ae9cb7546f4a2bcd for earlier term 13. Current term is 14. Ops: []
I20251212 03:12:29.385794 21344 consensus_queue.cc:1059] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 }, Status: INVALID_TERM, Last received: 13.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
W20251212 03:12:29.385882 21128 heartbeater.cc:646] Failed to heartbeat to 127.19.253.254:39701 (0 consecutive failures): Remote error: Failed to send heartbeat to master: Service unavailable: service kudu.master.MasterService not registered on Master
I20251212 03:12:29.385896 21344 raft_consensus.cc:3055] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 13 LEADER]: Stepping down as leader of term 13
I20251212 03:12:29.385946 21344 raft_consensus.cc:740] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 95ad0197220142e8ae9cb7546f4a2bcd, State: Running, Role: LEADER
I20251212 03:12:29.386001 21344 consensus_queue.cc:260] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 13.22, Last appended by leader: 22, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:29.386076 21344 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 13 FOLLOWER]: Advancing to term 14
I20251212 03:12:29.481005 21353 raft_consensus.cc:493] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 13 FOLLOWER]: Starting leader election (detected failure of leader 95ad0197220142e8ae9cb7546f4a2bcd)
I20251212 03:12:29.481124 21353 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 13 FOLLOWER]: Advancing to term 14
I20251212 03:12:29.481840 21353 raft_consensus.cc:515] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:29.482016 21353 leader_election.cc:290] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [CANDIDATE]: Term 14 election: Requested vote from peers 95ad0197220142e8ae9cb7546f4a2bcd (127.19.253.193:36423), 80f56db926b8409480a8452b3a121f58 (127.19.253.194:33281)
I20251212 03:12:29.482277 21178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" candidate_term: 14 candidate_status { last_received { term: 13 index: 22 } } ignore_live_leader: false dest_uuid: "80f56db926b8409480a8452b3a121f58"
I20251212 03:12:29.482277 21102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" candidate_term: 14 candidate_status { last_received { term: 13 index: 22 } } ignore_live_leader: false dest_uuid: "95ad0197220142e8ae9cb7546f4a2bcd"
I20251212 03:12:29.482414 21178 raft_consensus.cc:2393] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 8ac58b6f4a0b43eaad688ad29d00523c in current term 14: Already voted for candidate 80f56db926b8409480a8452b3a121f58 in this term.
I20251212 03:12:29.482950 21102 raft_consensus.cc:2468] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ac58b6f4a0b43eaad688ad29d00523c in term 14.
I20251212 03:12:29.483218 21219 leader_election.cc:304] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8ac58b6f4a0b43eaad688ad29d00523c, 95ad0197220142e8ae9cb7546f4a2bcd; no voters: 80f56db926b8409480a8452b3a121f58
I20251212 03:12:29.483351 21353 raft_consensus.cc:2804] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 14 FOLLOWER]: Leader election won for term 14
I20251212 03:12:29.483420 21353 raft_consensus.cc:697] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 14 LEADER]: Becoming Leader. State: Replica: 8ac58b6f4a0b43eaad688ad29d00523c, State: Running, Role: LEADER
I20251212 03:12:29.483481 21353 consensus_queue.cc:237] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 13.22, Last appended by leader: 22, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:29.495741 21350 raft_consensus.cc:493] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:29.495822 21350 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 14 FOLLOWER]: Advancing to term 15
I20251212 03:12:29.496461 21350 raft_consensus.cc:515] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:29.496603 21350 leader_election.cc:290] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [CANDIDATE]: Term 15 election: Requested vote from peers 95ad0197220142e8ae9cb7546f4a2bcd (127.19.253.193:36423), 8ac58b6f4a0b43eaad688ad29d00523c (127.19.253.195:40203)
I20251212 03:12:29.496798 21253 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "80f56db926b8409480a8452b3a121f58" candidate_term: 15 candidate_status { last_received { term: 13 index: 22 } } ignore_live_leader: false dest_uuid: "8ac58b6f4a0b43eaad688ad29d00523c"
I20251212 03:12:29.496804 21102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc80f8d16dd043b187b09967fc93dbe3" candidate_uuid: "80f56db926b8409480a8452b3a121f58" candidate_term: 15 candidate_status { last_received { term: 13 index: 22 } } ignore_live_leader: false dest_uuid: "95ad0197220142e8ae9cb7546f4a2bcd"
I20251212 03:12:29.496889 21102 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 14 FOLLOWER]: Advancing to term 15
I20251212 03:12:29.497411 21102 raft_consensus.cc:2468] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80f56db926b8409480a8452b3a121f58 in term 15.
I20251212 03:12:29.497602 21144 leader_election.cc:304] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 80f56db926b8409480a8452b3a121f58, 95ad0197220142e8ae9cb7546f4a2bcd; no voters: 8ac58b6f4a0b43eaad688ad29d00523c
I20251212 03:12:29.497704 21350 raft_consensus.cc:2804] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 15 FOLLOWER]: Leader election won for term 15
I20251212 03:12:29.497761 21350 raft_consensus.cc:697] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 15 LEADER]: Becoming Leader. State: Replica: 80f56db926b8409480a8452b3a121f58, State: Running, Role: LEADER
I20251212 03:12:29.497838 21350 consensus_queue.cc:237] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 13.22, Last appended by leader: 22, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:29.503886 20471 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 8e76e3d2bb9442aebce83fe50ed3349b [term 1 LEADER]: Raft consensus shutting down.
I20251212 03:12:29.503957 20471 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 8e76e3d2bb9442aebce83fe50ed3349b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251212 03:12:29.503975 20471 tablet_replica.cc:333] T 00000000000000000000000000000000 P 8e76e3d2bb9442aebce83fe50ed3349b: stopping tablet replica
I20251212 03:12:29.515986 20471 master.cc:583] Master@127.19.253.254:39701 shutdown complete.
I20251212 03:12:29.517421 20471 tablet_server.cc:178] TabletServer@127.19.253.193:0 shutting down...
I20251212 03:12:29.565776 20471 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251212 03:12:29.565959 20471 tablet_replica.cc:333] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd: stopping tablet replica
I20251212 03:12:29.566048 20471 raft_consensus.cc:2243] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 15 FOLLOWER]: Raft consensus shutting down.
I20251212 03:12:29.566116 20471 raft_consensus.cc:2272] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 15 FOLLOWER]: Raft consensus is shut down!
I20251212 03:12:29.567999 20471 tablet_server.cc:195] TabletServer@127.19.253.193:0 shutdown complete.
I20251212 03:12:29.568835 20471 tablet_server.cc:178] TabletServer@127.19.253.194:0 shutting down...
W20251212 03:12:29.570080 21217 consensus_peers.cc:597] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c -> Peer 80f56db926b8409480a8452b3a121f58 (127.19.253.194:33281): Couldn't send request to peer 80f56db926b8409480a8452b3a121f58. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20251212 03:12:29.592787 20471 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251212 03:12:29.592923 20471 tablet_replica.cc:333] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58: stopping tablet replica
I20251212 03:12:29.592983 20471 raft_consensus.cc:2243] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 15 LEADER]: Raft consensus shutting down.
I20251212 03:12:29.593053 20471 pending_rounds.cc:70] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58: Trying to abort 1 pending ops.
I20251212 03:12:29.593079 20471 pending_rounds.cc:77] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58: Aborting op as it isn't in flight: id { term: 15 index: 23 } timestamp: 7231525476343631872 op_type: NO_OP noop_request { }
I20251212 03:12:29.593140 20471 raft_consensus.cc:2889] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 15 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251212 03:12:29.593174 20471 raft_consensus.cc:2272] T bc80f8d16dd043b187b09967fc93dbe3 P 80f56db926b8409480a8452b3a121f58 [term 15 FOLLOWER]: Raft consensus is shut down!
W20251212 03:12:29.599915 21219 consensus_peers.cc:597] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c -> Peer 95ad0197220142e8ae9cb7546f4a2bcd (127.19.253.193:36423): Couldn't send request to peer 95ad0197220142e8ae9cb7546f4a2bcd. Status: Network error: Client connection negotiation failed: client connection to 127.19.253.193:36423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251212 03:12:29.605034 20471 tablet_server.cc:195] TabletServer@127.19.253.194:0 shutdown complete.
I20251212 03:12:29.605871 20471 tablet_server.cc:178] TabletServer@127.19.253.195:0 shutting down...
I20251212 03:12:29.609570 20471 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251212 03:12:29.609681 20471 tablet_replica.cc:333] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c: stopping tablet replica
I20251212 03:12:29.609731 20471 raft_consensus.cc:2243] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 14 LEADER]: Raft consensus shutting down.
I20251212 03:12:29.609802 20471 pending_rounds.cc:70] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c: Trying to abort 1 pending ops.
I20251212 03:12:29.609827 20471 pending_rounds.cc:77] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c: Aborting op as it isn't in flight: id { term: 14 index: 23 } timestamp: 7231525476284870656 op_type: NO_OP noop_request { }
I20251212 03:12:29.609882 20471 raft_consensus.cc:2889] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 14 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251212 03:12:29.609918 20471 raft_consensus.cc:2272] T bc80f8d16dd043b187b09967fc93dbe3 P 8ac58b6f4a0b43eaad688ad29d00523c [term 14 FOLLOWER]: Raft consensus is shut down!
I20251212 03:12:29.611541 20471 tablet_server.cc:195] TabletServer@127.19.253.195:0 shutdown complete.
I20251212 03:12:29.612268 20471 timestamp_advancement-itest.cc:282] Restarting single tablet server...
I20251212 03:12:29.612385 20471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 03:12:29.613651 21359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:29.613687 21357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:29.613684 21356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:12:29.613936 20471 server_base.cc:1047] running on GCE node
I20251212 03:12:29.614045 20471 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251212 03:12:29.614078 20471 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251212 03:12:29.614102 20471 hybrid_clock.cc:648] HybridClock initialized: now 1765509149614101 us; error 0 us; skew 500 ppm
I20251212 03:12:29.614647 20471 webserver.cc:492] Webserver started at http://127.19.253.193:46603/ using document root <none> and password file <none>
I20251212 03:12:29.614730 20471 fs_manager.cc:362] Metadata directory not provided
I20251212 03:12:29.614766 20471 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251212 03:12:29.615424 20471 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20251212 03:12:29.616710 21366 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:29.616847 20471 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251212 03:12:29.616973 20471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765509114134682-20471-0/minicluster-data/ts-0-root
uuid: "95ad0197220142e8ae9cb7546f4a2bcd"
format_stamp: "Formatted at 2025-12-12 03:12:27 on dist-test-slave-q05w"
I20251212 03:12:29.617125 20471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765509114134682-20471-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765509114134682-20471-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1765509114134682-20471-0/minicluster-data/ts-0-root/data
Total live blocks: 5
Total live bytes: 5320
Total live bytes (after alignment): 24576
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 03:12:29.632793 20471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 03:12:29.632990 20471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 03:12:29.633140 20471 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251212 03:12:29.633419 21373 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20251212 03:12:29.634104 20471 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251212 03:12:29.634153 20471 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251212 03:12:29.634337 20471 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20251212 03:12:29.634768 20471 ts_tablet_manager.cc:616] Registered 1 tablets
I20251212 03:12:29.634799 20471 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:29.634811 21373 tablet_bootstrap.cc:492] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd: Bootstrap starting.
I20251212 03:12:29.636499 21373 tablet_bootstrap.cc:492] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251212 03:12:29.636806 21373 tablet_bootstrap.cc:492] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd: Bootstrap complete.
I20251212 03:12:29.636925 21373 ts_tablet_manager.cc:1403] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251212 03:12:29.637066 21373 raft_consensus.cc:359] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 15 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:29.637152 21373 raft_consensus.cc:740] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 15 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95ad0197220142e8ae9cb7546f4a2bcd, State: Initialized, Role: FOLLOWER
I20251212 03:12:29.637213 21373 consensus_queue.cc:260] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 22, Last appended: 13.22, Last appended by leader: 22, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:29.637398 21373 ts_tablet_manager.cc:1434] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20251212 03:12:29.638043 20471 rpc_server.cc:307] RPC server started. Bound to: 127.19.253.193:36423
I20251212 03:12:29.640089 21432 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.253.193:36423 every 8 connection(s)
W20251212 03:12:29.640836 21433 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20251212 03:12:29.646731 20471 tablet_server.cc:178] TabletServer@127.19.253.193:36423 shutting down...
I20251212 03:12:29.757355 21419 raft_consensus.cc:493] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:29.757473 21419 raft_consensus.cc:3060] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 15 FOLLOWER]: Advancing to term 16
I20251212 03:12:29.758271 21419 raft_consensus.cc:515] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ad0197220142e8ae9cb7546f4a2bcd" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 36423 } } peers { permanent_uuid: "80f56db926b8409480a8452b3a121f58" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 33281 } } peers { permanent_uuid: "8ac58b6f4a0b43eaad688ad29d00523c" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 40203 } }
I20251212 03:12:29.758497 21419 leader_election.cc:290] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [CANDIDATE]: Term 16 election: Requested vote from peers 80f56db926b8409480a8452b3a121f58 (127.19.253.194:33281), 8ac58b6f4a0b43eaad688ad29d00523c (127.19.253.195:40203)
W20251212 03:12:29.759187 21368 leader_election.cc:336] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 80f56db926b8409480a8452b3a121f58 (127.19.253.194:33281): Network error: Client connection negotiation failed: client connection to 127.19.253.194:33281: connect: Connection refused (error 111)
W20251212 03:12:29.759245 21370 leader_election.cc:336] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 8ac58b6f4a0b43eaad688ad29d00523c (127.19.253.195:40203): Network error: Client connection negotiation failed: client connection to 127.19.253.195:40203: connect: Connection refused (error 111)
I20251212 03:12:29.759274 21368 leader_election.cc:304] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 95ad0197220142e8ae9cb7546f4a2bcd; no voters: 80f56db926b8409480a8452b3a121f58, 8ac58b6f4a0b43eaad688ad29d00523c
I20251212 03:12:29.759356 21419 raft_consensus.cc:2749] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20251212 03:12:29.890618 20471 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251212 03:12:29.890808 20471 tablet_replica.cc:333] stopping tablet replica
I20251212 03:12:29.890887 20471 raft_consensus.cc:2243] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 16 FOLLOWER]: Raft consensus shutting down.
I20251212 03:12:29.890951 20471 raft_consensus.cc:2272] T bc80f8d16dd043b187b09967fc93dbe3 P 95ad0197220142e8ae9cb7546f4a2bcd [term 16 FOLLOWER]: Raft consensus is shut down!
I20251212 03:12:29.903048 20471 tablet_server.cc:195] TabletServer@127.19.253.193:36423 shutdown complete.
[ OK ] TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap (2940 ms)
[ RUN ] TimestampAdvancementITest.Kudu2463Test
I20251212 03:12:29.907827 20471 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.253.254:46103
I20251212 03:12:29.907991 20471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 03:12:29.909350 21441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:12:29.909610 20471 server_base.cc:1047] running on GCE node
W20251212 03:12:29.909531 21443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:29.909533 21440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:12:29.909847 20471 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251212 03:12:29.909888 20471 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251212 03:12:29.909900 20471 hybrid_clock.cc:648] HybridClock initialized: now 1765509149909900 us; error 0 us; skew 500 ppm
I20251212 03:12:29.910477 20471 webserver.cc:492] Webserver started at http://127.19.253.254:37785/ using document root <none> and password file <none>
I20251212 03:12:29.910552 20471 fs_manager.cc:362] Metadata directory not provided
I20251212 03:12:29.910585 20471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 03:12:29.910624 20471 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251212 03:12:29.910852 20471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765509114134682-20471-0/minicluster-data/master-0-root/instance:
uuid: "55fddf801b49476d88fa2f1a4842ce0e"
format_stamp: "Formatted at 2025-12-12 03:12:29 on dist-test-slave-q05w"
I20251212 03:12:29.911692 20471 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251212 03:12:29.912204 21448 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:29.912324 20471 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:29.912386 20471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765509114134682-20471-0/minicluster-data/master-0-root
uuid: "55fddf801b49476d88fa2f1a4842ce0e"
format_stamp: "Formatted at 2025-12-12 03:12:29 on dist-test-slave-q05w"
I20251212 03:12:29.912429 20471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765509114134682-20471-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765509114134682-20471-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765509114134682-20471-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 03:12:29.919669 20471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 03:12:29.919842 20471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 03:12:29.922326 20471 rpc_server.cc:307] RPC server started. Bound to: 127.19.253.254:46103
I20251212 03:12:29.922912 21500 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.253.254:46103 every 8 connection(s)
I20251212 03:12:29.923059 21501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 03:12:29.924031 21501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e: Bootstrap starting.
I20251212 03:12:29.924353 21501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e: Neither blocks nor log segments found. Creating new log.
I20251212 03:12:29.924854 21501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e: No bootstrap required, opened a new log
I20251212 03:12:29.924989 21501 raft_consensus.cc:359] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fddf801b49476d88fa2f1a4842ce0e" member_type: VOTER }
I20251212 03:12:29.925045 21501 raft_consensus.cc:385] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 03:12:29.925068 21501 raft_consensus.cc:740] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fddf801b49476d88fa2f1a4842ce0e, State: Initialized, Role: FOLLOWER
I20251212 03:12:29.925117 21501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fddf801b49476d88fa2f1a4842ce0e" member_type: VOTER }
I20251212 03:12:29.925160 21501 raft_consensus.cc:399] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251212 03:12:29.925185 21501 raft_consensus.cc:493] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251212 03:12:29.925212 21501 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e [term 0 FOLLOWER]: Advancing to term 1
I20251212 03:12:29.925700 21501 raft_consensus.cc:515] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fddf801b49476d88fa2f1a4842ce0e" member_type: VOTER }
I20251212 03:12:29.925761 21501 leader_election.cc:304] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55fddf801b49476d88fa2f1a4842ce0e; no voters:
I20251212 03:12:29.925863 21501 leader_election.cc:290] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20251212 03:12:29.925988 21501 sys_catalog.cc:565] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e [sys.catalog]: configured and running, proceeding with master startup.
I20251212 03:12:29.927256 20471 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251212 03:12:29.930286 21504 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e [term 1 FOLLOWER]: Leader election won for term 1
I20251212 03:12:29.930405 21504 raft_consensus.cc:697] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e [term 1 LEADER]: Becoming Leader. State: Replica: 55fddf801b49476d88fa2f1a4842ce0e, State: Running, Role: LEADER
I20251212 03:12:29.930450 21504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fddf801b49476d88fa2f1a4842ce0e" member_type: VOTER }
I20251212 03:12:29.930615 21504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55fddf801b49476d88fa2f1a4842ce0e. Latest consensus state: current_term: 1 leader_uuid: "55fddf801b49476d88fa2f1a4842ce0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fddf801b49476d88fa2f1a4842ce0e" member_type: VOTER } }
I20251212 03:12:29.930652 21517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55fddf801b49476d88fa2f1a4842ce0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55fddf801b49476d88fa2f1a4842ce0e" member_type: VOTER } }
I20251212 03:12:29.930732 21517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e [sys.catalog]: This master's current role is: LEADER
W20251212 03:12:29.930739 21516 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251212 03:12:29.931023 21516 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251212 03:12:29.931156 21519 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251212 03:12:29.931272 21519 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251212 03:12:29.931811 21519 catalog_manager.cc:1357] Generated new cluster ID: 319ddae6e83f4ab6842fc52a8a698de3
I20251212 03:12:29.931854 21519 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251212 03:12:29.930657 21504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e [sys.catalog]: This master's current role is: LEADER
I20251212 03:12:29.949055 21519 catalog_manager.cc:1380] Generated new certificate authority record
I20251212 03:12:29.949468 21519 catalog_manager.cc:1514] Loading token signing keys...
I20251212 03:12:29.957088 21519 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e: Generated new TSK 0
I20251212 03:12:29.957165 21519 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251212 03:12:29.959110 20471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 03:12:29.960547 21525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:29.960759 21523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:29.960778 21522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:12:29.960839 20471 server_base.cc:1047] running on GCE node
I20251212 03:12:29.960994 20471 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251212 03:12:29.961030 20471 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251212 03:12:29.961043 20471 hybrid_clock.cc:648] HybridClock initialized: now 1765509149961043 us; error 0 us; skew 500 ppm
I20251212 03:12:29.961687 20471 webserver.cc:492] Webserver started at http://127.19.253.193:36677/ using document root <none> and password file <none>
I20251212 03:12:29.961769 20471 fs_manager.cc:362] Metadata directory not provided
I20251212 03:12:29.961817 20471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 03:12:29.961866 20471 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251212 03:12:29.962116 20471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765509114134682-20471-0/minicluster-data/ts-0-root/instance:
uuid: "6718802d9d1b4ea0b2297512a373ad5e"
format_stamp: "Formatted at 2025-12-12 03:12:29 on dist-test-slave-q05w"
I20251212 03:12:29.963028 20471 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251212 03:12:29.963515 21530 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:29.963620 20471 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:29.963666 20471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765509114134682-20471-0/minicluster-data/ts-0-root
uuid: "6718802d9d1b4ea0b2297512a373ad5e"
format_stamp: "Formatted at 2025-12-12 03:12:29 on dist-test-slave-q05w"
I20251212 03:12:29.963712 20471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765509114134682-20471-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765509114134682-20471-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765509114134682-20471-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 03:12:29.977841 20471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 03:12:29.978035 20471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 03:12:29.978191 20471 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251212 03:12:29.978437 20471 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251212 03:12:29.978480 20471 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:29.978515 20471 ts_tablet_manager.cc:616] Registered 0 tablets
I20251212 03:12:29.978541 20471 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:29.981829 20471 rpc_server.cc:307] RPC server started. Bound to: 127.19.253.193:41509
I20251212 03:12:29.981891 21593 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.253.193:41509 every 8 connection(s)
I20251212 03:12:29.982192 20471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 03:12:29.983459 21599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:29.983508 21598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:29.983702 21601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:12:29.984354 20471 server_base.cc:1047] running on GCE node
I20251212 03:12:29.984503 20471 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251212 03:12:29.984539 20471 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251212 03:12:29.984556 20471 hybrid_clock.cc:648] HybridClock initialized: now 1765509149984556 us; error 0 us; skew 500 ppm
I20251212 03:12:29.985176 20471 webserver.cc:492] Webserver started at http://127.19.253.194:33515/ using document root <none> and password file <none>
I20251212 03:12:29.985255 20471 fs_manager.cc:362] Metadata directory not provided
I20251212 03:12:29.985288 20471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 03:12:29.985342 20471 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251212 03:12:29.985663 20471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765509114134682-20471-0/minicluster-data/ts-1-root/instance:
uuid: "3ecce86d24b04d3cae917bf243a4d819"
format_stamp: "Formatted at 2025-12-12 03:12:29 on dist-test-slave-q05w"
I20251212 03:12:29.985883 21594 heartbeater.cc:344] Connected to a master server at 127.19.253.254:46103
I20251212 03:12:29.985960 21594 heartbeater.cc:461] Registering TS with master...
I20251212 03:12:29.986068 21594 heartbeater.cc:507] Master 127.19.253.254:46103 requested a full tablet report, sending...
I20251212 03:12:29.986333 21465 ts_manager.cc:194] Registered new tserver with Master: 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193:41509)
I20251212 03:12:29.986817 20471 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251212 03:12:29.987078 21465 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45898
I20251212 03:12:29.987425 21606 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:29.987600 20471 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:29.987644 20471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765509114134682-20471-0/minicluster-data/ts-1-root
uuid: "3ecce86d24b04d3cae917bf243a4d819"
format_stamp: "Formatted at 2025-12-12 03:12:29 on dist-test-slave-q05w"
I20251212 03:12:29.987687 20471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765509114134682-20471-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765509114134682-20471-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765509114134682-20471-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 03:12:29.997287 20471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 03:12:29.997499 20471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 03:12:29.997644 20471 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251212 03:12:29.997831 20471 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251212 03:12:29.997859 20471 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:29.997892 20471 ts_tablet_manager.cc:616] Registered 0 tablets
I20251212 03:12:29.997920 20471 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:30.000878 20471 rpc_server.cc:307] RPC server started. Bound to: 127.19.253.194:43843
I20251212 03:12:30.000952 21669 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.253.194:43843 every 8 connection(s)
I20251212 03:12:30.001224 20471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 03:12:30.002676 21676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:30.002691 21673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:30.002822 21674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:12:30.003003 20471 server_base.cc:1047] running on GCE node
I20251212 03:12:30.003129 20471 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251212 03:12:30.003170 20471 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251212 03:12:30.003197 20471 hybrid_clock.cc:648] HybridClock initialized: now 1765509150003198 us; error 0 us; skew 500 ppm
I20251212 03:12:30.003865 20471 webserver.cc:492] Webserver started at http://127.19.253.195:42473/ using document root <none> and password file <none>
I20251212 03:12:30.003962 20471 fs_manager.cc:362] Metadata directory not provided
I20251212 03:12:30.004000 20471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 03:12:30.004030 20471 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251212 03:12:30.004328 20471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765509114134682-20471-0/minicluster-data/ts-2-root/instance:
uuid: "9983e22759b64f91ade67eb78d5ce421"
format_stamp: "Formatted at 2025-12-12 03:12:30 on dist-test-slave-q05w"
I20251212 03:12:30.004745 21670 heartbeater.cc:344] Connected to a master server at 127.19.253.254:46103
I20251212 03:12:30.004799 21670 heartbeater.cc:461] Registering TS with master...
I20251212 03:12:30.004900 21670 heartbeater.cc:507] Master 127.19.253.254:46103 requested a full tablet report, sending...
I20251212 03:12:30.005143 21465 ts_manager.cc:194] Registered new tserver with Master: 3ecce86d24b04d3cae917bf243a4d819 (127.19.253.194:43843)
I20251212 03:12:30.005534 20471 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251212 03:12:30.005582 21465 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45906
I20251212 03:12:30.006094 21681 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:30.006258 20471 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:30.006311 20471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765509114134682-20471-0/minicluster-data/ts-2-root
uuid: "9983e22759b64f91ade67eb78d5ce421"
format_stamp: "Formatted at 2025-12-12 03:12:30 on dist-test-slave-q05w"
I20251212 03:12:30.006342 20471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765509114134682-20471-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765509114134682-20471-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765509114134682-20471-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 03:12:30.022135 20471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 03:12:30.022475 20471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 03:12:30.022639 20471 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251212 03:12:30.022840 20471 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251212 03:12:30.022868 20471 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:30.022894 20471 ts_tablet_manager.cc:616] Registered 0 tablets
I20251212 03:12:30.022924 20471 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:30.026108 20471 rpc_server.cc:307] RPC server started. Bound to: 127.19.253.195:36039
I20251212 03:12:30.026187 21744 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.253.195:36039 every 8 connection(s)
I20251212 03:12:30.029613 21745 heartbeater.cc:344] Connected to a master server at 127.19.253.254:46103
I20251212 03:12:30.029678 21745 heartbeater.cc:461] Registering TS with master...
I20251212 03:12:30.029789 21745 heartbeater.cc:507] Master 127.19.253.254:46103 requested a full tablet report, sending...
I20251212 03:12:30.030078 21465 ts_manager.cc:194] Registered new tserver with Master: 9983e22759b64f91ade67eb78d5ce421 (127.19.253.195:36039)
I20251212 03:12:30.030474 21465 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45918
I20251212 03:12:30.030721 20471 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004389406s
I20251212 03:12:30.038002 20471 test_util.cc:276] Using random seed: -1581492240
I20251212 03:12:30.042383 21465 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45936:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251212 03:12:30.042526 21465 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251212 03:12:30.047094 21558 tablet_service.cc:1505] Processing CreateTablet for tablet fe0a5b95bc99492e85e90eb6bdfe425b (DEFAULT_TABLE table=test-workload [id=caace4758ef54dad80fb987e63392cba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 03:12:30.047274 21558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe0a5b95bc99492e85e90eb6bdfe425b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 03:12:30.047549 21634 tablet_service.cc:1505] Processing CreateTablet for tablet fe0a5b95bc99492e85e90eb6bdfe425b (DEFAULT_TABLE table=test-workload [id=caace4758ef54dad80fb987e63392cba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 03:12:30.047653 21634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe0a5b95bc99492e85e90eb6bdfe425b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 03:12:30.047684 21709 tablet_service.cc:1505] Processing CreateTablet for tablet fe0a5b95bc99492e85e90eb6bdfe425b (DEFAULT_TABLE table=test-workload [id=caace4758ef54dad80fb987e63392cba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 03:12:30.047787 21709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe0a5b95bc99492e85e90eb6bdfe425b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 03:12:30.048882 21767 tablet_bootstrap.cc:492] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421: Bootstrap starting.
I20251212 03:12:30.049132 21768 tablet_bootstrap.cc:492] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819: Bootstrap starting.
I20251212 03:12:30.049365 21767 tablet_bootstrap.cc:654] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421: Neither blocks nor log segments found. Creating new log.
I20251212 03:12:30.049592 21768 tablet_bootstrap.cc:654] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819: Neither blocks nor log segments found. Creating new log.
I20251212 03:12:30.049758 21766 tablet_bootstrap.cc:492] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e: Bootstrap starting.
I20251212 03:12:30.049957 21767 tablet_bootstrap.cc:492] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421: No bootstrap required, opened a new log
I20251212 03:12:30.050005 21767 ts_tablet_manager.cc:1403] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251212 03:12:30.050161 21766 tablet_bootstrap.cc:654] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e: Neither blocks nor log segments found. Creating new log.
I20251212 03:12:30.050163 21767 raft_consensus.cc:359] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.050303 21767 raft_consensus.cc:385] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 03:12:30.050333 21767 raft_consensus.cc:740] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9983e22759b64f91ade67eb78d5ce421, State: Initialized, Role: FOLLOWER
I20251212 03:12:30.050395 21767 consensus_queue.cc:260] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.050568 21767 ts_tablet_manager.cc:1434] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251212 03:12:30.050612 21745 heartbeater.cc:499] Master 127.19.253.254:46103 was elected leader, sending a full tablet report...
I20251212 03:12:30.050698 21768 tablet_bootstrap.cc:492] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819: No bootstrap required, opened a new log
I20251212 03:12:30.050753 21768 ts_tablet_manager.cc:1403] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251212 03:12:30.050817 21766 tablet_bootstrap.cc:492] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e: No bootstrap required, opened a new log
I20251212 03:12:30.050864 21766 ts_tablet_manager.cc:1403] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251212 03:12:30.050923 21768 raft_consensus.cc:359] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.051000 21768 raft_consensus.cc:385] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 03:12:30.051028 21768 raft_consensus.cc:740] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ecce86d24b04d3cae917bf243a4d819, State: Initialized, Role: FOLLOWER
I20251212 03:12:30.051026 21766 raft_consensus.cc:359] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.051107 21768 consensus_queue.cc:260] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.051131 21766 raft_consensus.cc:385] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 03:12:30.051261 21766 raft_consensus.cc:740] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6718802d9d1b4ea0b2297512a373ad5e, State: Initialized, Role: FOLLOWER
I20251212 03:12:30.051292 21768 ts_tablet_manager.cc:1434] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251212 03:12:30.051373 21670 heartbeater.cc:499] Master 127.19.253.254:46103 was elected leader, sending a full tablet report...
I20251212 03:12:30.051319 21766 consensus_queue.cc:260] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.051540 21766 ts_tablet_manager.cc:1434] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251212 03:12:30.051584 21594 heartbeater.cc:499] Master 127.19.253.254:46103 was elected leader, sending a full tablet report...
I20251212 03:12:30.308722 21773 raft_consensus.cc:493] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251212 03:12:30.308831 21773 raft_consensus.cc:515] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.309003 21773 leader_election.cc:290] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9983e22759b64f91ade67eb78d5ce421 (127.19.253.195:36039), 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193:41509)
I20251212 03:12:30.312355 21719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe0a5b95bc99492e85e90eb6bdfe425b" candidate_uuid: "3ecce86d24b04d3cae917bf243a4d819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9983e22759b64f91ade67eb78d5ce421" is_pre_election: true
I20251212 03:12:30.312364 21568 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe0a5b95bc99492e85e90eb6bdfe425b" candidate_uuid: "3ecce86d24b04d3cae917bf243a4d819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6718802d9d1b4ea0b2297512a373ad5e" is_pre_election: true
I20251212 03:12:30.312470 21719 raft_consensus.cc:2468] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ecce86d24b04d3cae917bf243a4d819 in term 0.
I20251212 03:12:30.312500 21568 raft_consensus.cc:2468] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ecce86d24b04d3cae917bf243a4d819 in term 0.
I20251212 03:12:30.312618 21610 leader_election.cc:304] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ecce86d24b04d3cae917bf243a4d819, 9983e22759b64f91ade67eb78d5ce421; no voters:
I20251212 03:12:30.312727 21773 raft_consensus.cc:2804] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251212 03:12:30.312774 21773 raft_consensus.cc:493] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251212 03:12:30.312791 21773 raft_consensus.cc:3060] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 0 FOLLOWER]: Advancing to term 1
I20251212 03:12:30.313464 21773 raft_consensus.cc:515] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.313585 21773 leader_election.cc:290] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [CANDIDATE]: Term 1 election: Requested vote from peers 9983e22759b64f91ade67eb78d5ce421 (127.19.253.195:36039), 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193:41509)
I20251212 03:12:30.313731 21719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe0a5b95bc99492e85e90eb6bdfe425b" candidate_uuid: "3ecce86d24b04d3cae917bf243a4d819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9983e22759b64f91ade67eb78d5ce421"
I20251212 03:12:30.313741 21568 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fe0a5b95bc99492e85e90eb6bdfe425b" candidate_uuid: "3ecce86d24b04d3cae917bf243a4d819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6718802d9d1b4ea0b2297512a373ad5e"
I20251212 03:12:30.313796 21719 raft_consensus.cc:3060] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 0 FOLLOWER]: Advancing to term 1
I20251212 03:12:30.313809 21568 raft_consensus.cc:3060] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 0 FOLLOWER]: Advancing to term 1
I20251212 03:12:30.314512 21719 raft_consensus.cc:2468] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ecce86d24b04d3cae917bf243a4d819 in term 1.
I20251212 03:12:30.314534 21568 raft_consensus.cc:2468] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ecce86d24b04d3cae917bf243a4d819 in term 1.
I20251212 03:12:30.314704 21610 leader_election.cc:304] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ecce86d24b04d3cae917bf243a4d819, 9983e22759b64f91ade67eb78d5ce421; no voters:
I20251212 03:12:30.314805 21773 raft_consensus.cc:2804] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 1 FOLLOWER]: Leader election won for term 1
I20251212 03:12:30.314951 21773 raft_consensus.cc:697] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 1 LEADER]: Becoming Leader. State: Replica: 3ecce86d24b04d3cae917bf243a4d819, State: Running, Role: LEADER
I20251212 03:12:30.315006 21773 consensus_queue.cc:237] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.315567 21464 catalog_manager.cc:5654] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ecce86d24b04d3cae917bf243a4d819 (127.19.253.194). New cstate: current_term: 1 leader_uuid: "3ecce86d24b04d3cae917bf243a4d819" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:30.329248 21568 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251212 03:12:30.329262 21719 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251212 03:12:30.329537 21776 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251212 03:12:30.329628 21773 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251212 03:12:30.346871 21644 consensus_queue.cc:237] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.347297 21719 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20251212 03:12:30.347316 21568 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20251212 03:12:30.347484 21776 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20251212 03:12:30.347575 21794 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20251212 03:12:30.348242 21773 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index -1 to 16, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.348433 21719 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index -1 to 16, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.349205 21464 catalog_manager.cc:5654] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 reported cstate change: config changed from index -1 to 16, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "3ecce86d24b04d3cae917bf243a4d819" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:30.349215 21568 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 LEARNER]: Committing config change with OpId 1.16: config changed from index -1 to 16, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.349747 21644 consensus_queue.cc:237] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
W20251212 03:12:30.349925 21464 catalog_manager.cc:5332] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fe0a5b95bc99492e85e90eb6bdfe425b with cas_config_opid_index 16: no extra replica candidate found for tablet fe0a5b95bc99492e85e90eb6bdfe425b (table test-workload [id=caace4758ef54dad80fb987e63392cba]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20251212 03:12:30.350423 21568 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 LEARNER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20251212 03:12:30.350543 21719 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20251212 03:12:30.350653 21794 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20251212 03:12:30.350739 21773 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20251212 03:12:30.351254 21776 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 1 LEADER]: Committing config change with OpId 1.17: config changed from index 16 to 17, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.351388 21719 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.351438 21568 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.352264 21465 catalog_manager.cc:5654] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 reported cstate change: config changed from index 16 to 17, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3ecce86d24b04d3cae917bf243a4d819" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:30.352751 21644 consensus_queue.cc:237] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.353212 21568 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20251212 03:12:30.353250 21719 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20251212 03:12:30.353415 21794 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20251212 03:12:30.353488 21773 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20251212 03:12:30.353982 21776 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 1 LEADER]: Committing config change with OpId 1.18: config changed from index 17 to 18, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.354136 21568 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 LEARNER]: Committing config change with OpId 1.18: config changed from index 17 to 18, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.354851 21719 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.354959 21463 catalog_manager.cc:5654] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 reported cstate change: config changed from index 17 to 18, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "3ecce86d24b04d3cae917bf243a4d819" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:30.354956 21644 consensus_queue.cc:237] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 1.18, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.355482 21567 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 LEARNER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
I20251212 03:12:30.355825 21718 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
I20251212 03:12:30.355844 21801 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20251212 03:12:30.356123 21801 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20251212 03:12:30.356462 21776 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 1 LEADER]: Committing config change with OpId 1.19: config changed from index 18 to 19, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.356603 21568 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index 18 to 19, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.356685 21718 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index 18 to 19, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.357631 21464 catalog_manager.cc:5654] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e reported cstate change: config changed from index 18 to 19, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3ecce86d24b04d3cae917bf243a4d819" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.357769 21644 consensus_queue.cc:237] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 1.19, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.358162 21718 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20251212 03:12:30.358325 21568 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20251212 03:12:30.358480 21801 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20251212 03:12:30.358564 21794 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20251212 03:12:30.358955 21801 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 1 LEADER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.359055 21718 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.359112 21568 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 LEARNER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.360018 21465 catalog_manager.cc:5654] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 reported cstate change: config changed from index 19 to 20, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "3ecce86d24b04d3cae917bf243a4d819" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:30.360065 21644 consensus_queue.cc:237] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 1.20, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.360422 21567 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 LEARNER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch)
I20251212 03:12:30.360499 21718 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch)
I20251212 03:12:30.360697 21794 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20251212 03:12:30.360792 21794 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20251212 03:12:30.361322 21776 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 1 LEADER]: Committing config change with OpId 1.21: config changed from index 20 to 21, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.361552 21718 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Committing config change with OpId 1.21: config changed from index 20 to 21, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.362066 21567 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Committing config change with OpId 1.21: config changed from index 20 to 21, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.362068 21464 catalog_manager.cc:5654] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 reported cstate change: config changed from index 20 to 21, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3ecce86d24b04d3cae917bf243a4d819" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:30.363235 21644 consensus_queue.cc:237] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 1.21, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.363592 21567 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 1 index: 22. (index mismatch)
I20251212 03:12:30.363593 21718 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 1 index: 22. (index mismatch)
I20251212 03:12:30.363796 21776 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20251212 03:12:30.363888 21773 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20251212 03:12:30.364404 21776 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 1 LEADER]: Committing config change with OpId 1.22: config changed from index 21 to 22, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.364552 21718 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Committing config change with OpId 1.22: config changed from index 21 to 22, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.365240 21463 catalog_manager.cc:5654] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 reported cstate change: config changed from index 21 to 22, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "3ecce86d24b04d3cae917bf243a4d819" committed_config { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:30.365379 21567 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 LEARNER]: Committing config change with OpId 1.22: config changed from index 21 to 22, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.366441 21644 consensus_queue.cc:237] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 1.22, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.366758 21718 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 1 index: 23. (index mismatch)
I20251212 03:12:30.366758 21567 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 LEARNER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 1 index: 23. (index mismatch)
I20251212 03:12:30.366930 21773 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20251212 03:12:30.367008 21801 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20251212 03:12:30.367511 21801 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 1 LEADER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.367623 21718 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.368191 21567 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.368395 21463 catalog_manager.cc:5654] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 reported cstate change: config changed from index 22 to 23, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3ecce86d24b04d3cae917bf243a4d819" committed_config { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:30.368599 21644 consensus_queue.cc:237] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.369068 21568 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20251212 03:12:30.369230 21719 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20251212 03:12:30.369374 21776 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20251212 03:12:30.369369 21451 catalog_manager.cc:5201] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fe0a5b95bc99492e85e90eb6bdfe425b with cas_config_opid_index 16: aborting the task: latest config opid_index 23; task opid_index 16
I20251212 03:12:30.369457 21801 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20251212 03:12:30.369935 21801 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 1 LEADER]: Committing config change with OpId 1.24: config changed from index 23 to 24, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.370034 21719 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index 23 to 24, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.370829 21463 catalog_manager.cc:5654] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 reported cstate change: config changed from index 23 to 24, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "3ecce86d24b04d3cae917bf243a4d819" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:30.370890 21568 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 LEARNER]: Committing config change with OpId 1.24: config changed from index 23 to 24, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.370988 21644 consensus_queue.cc:237] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.371640 21719 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20251212 03:12:30.371750 21567 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 LEARNER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20251212 03:12:30.371896 21801 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20251212 03:12:30.371963 21776 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20251212 03:12:30.372484 21773 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.372855 21568 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.373430 21719 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.373479 21644 consensus_queue.cc:237] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.373464 21463 catalog_manager.cc:5654] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 reported cstate change: config changed from index 24 to 25, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3ecce86d24b04d3cae917bf243a4d819" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:30.373984 21568 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20251212 03:12:30.374120 21776 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20251212 03:12:30.374711 21718 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20251212 03:12:30.374892 21794 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20251212 03:12:30.375385 21801 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.375502 21718 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.376222 21465 catalog_manager.cc:5654] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 reported cstate change: config changed from index 25 to 26, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "3ecce86d24b04d3cae917bf243a4d819" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } health_report { overall_health: UNKNOWN } } }
I20251212 03:12:30.376416 21567 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 LEARNER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.377143 21644 consensus_queue.cc:237] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.377257 21450 catalog_manager.cc:5201] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fe0a5b95bc99492e85e90eb6bdfe425b with cas_config_opid_index 18: aborting the task: latest config opid_index 26; task opid_index 18
I20251212 03:12:30.377537 21718 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20251212 03:12:30.377565 21567 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 LEARNER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20251212 03:12:30.377712 21776 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251212 03:12:30.377803 21801 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251212 03:12:30.378417 21773 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.378566 21567 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.378538 21718 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.379365 21465 catalog_manager.cc:5654] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 reported cstate change: config changed from index 26 to 27, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3ecce86d24b04d3cae917bf243a4d819" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:30.379711 21644 consensus_queue.cc:237] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.380107 21567 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20251212 03:12:30.380247 21718 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20251212 03:12:30.380259 21776 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251212 03:12:30.380403 21776 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251212 03:12:30.380913 21773 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.381047 21718 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.381822 21464 catalog_manager.cc:5654] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 reported cstate change: config changed from index 27 to 28, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "3ecce86d24b04d3cae917bf243a4d819" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.381927 21567 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 LEARNER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.382810 21644 consensus_queue.cc:237] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.383167 21567 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 LEARNER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20251212 03:12:30.383183 21718 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20251212 03:12:30.383363 21794 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251212 03:12:30.383427 21773 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251212 03:12:30.384012 21801 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.384137 21567 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.384186 21718 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.384883 21464 catalog_manager.cc:5654] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e reported cstate change: config changed from index 28 to 29, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3ecce86d24b04d3cae917bf243a4d819" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.385529 21644 consensus_queue.cc:237] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.385937 21718 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20251212 03:12:30.385999 21567 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20251212 03:12:30.386075 21773 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251212 03:12:30.386161 21776 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251212 03:12:30.386615 21773 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.386765 21718 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.386998 21567 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 LEARNER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.387668 21463 catalog_manager.cc:5654] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 reported cstate change: config changed from index 29 to 30, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "3ecce86d24b04d3cae917bf243a4d819" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:30.387899 21644 consensus_queue.cc:237] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.388240 21567 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 LEARNER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20251212 03:12:30.388484 21801 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251212 03:12:30.388494 21718 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20251212 03:12:30.388671 21801 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251212 03:12:30.389074 21776 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.389259 21567 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.389503 21718 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.390079 21465 catalog_manager.cc:5654] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 reported cstate change: config changed from index 30 to 31, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3ecce86d24b04d3cae917bf243a4d819" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:30.390141 21644 consensus_queue.cc:237] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.390599 21718 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20251212 03:12:30.390633 21568 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20251212 03:12:30.390784 21801 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251212 03:12:30.391090 21776 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251212 03:12:30.391512 21801 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index 31 to 32, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.391676 21567 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 LEARNER]: Committing config change with OpId 1.32: config changed from index 31 to 32, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.391666 21718 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.392519 21463 catalog_manager.cc:5654] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 reported cstate change: config changed from index 31 to 32, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "3ecce86d24b04d3cae917bf243a4d819" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:30.392846 21644 consensus_queue.cc:237] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.393201 21718 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20251212 03:12:30.393384 21567 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 LEARNER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20251212 03:12:30.393463 21801 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251212 03:12:30.393651 21801 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251212 03:12:30.393893 21776 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 1 LEADER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.393998 21718 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.394311 21567 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.394838 21464 catalog_manager.cc:5654] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 reported cstate change: config changed from index 32 to 33, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3ecce86d24b04d3cae917bf243a4d819" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:30.395152 21644 consensus_queue.cc:237] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 1.33, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.395638 21567 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20251212 03:12:30.395676 21718 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20251212 03:12:30.395807 21801 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251212 03:12:30.395901 21794 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251212 03:12:30.396382 21776 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 1 LEADER]: Committing config change with OpId 1.34: config changed from index 33 to 34, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.396481 21718 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index 33 to 34, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.397133 21567 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 LEARNER]: Committing config change with OpId 1.34: config changed from index 33 to 34, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.397243 21463 catalog_manager.cc:5654] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 reported cstate change: config changed from index 33 to 34, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "3ecce86d24b04d3cae917bf243a4d819" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: NON_VOTER last_known_addr { host: "127.19.253.193" port: 41509 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:30.397289 21644 consensus_queue.cc:237] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 1.34, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.398105 21568 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 LEARNER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20251212 03:12:30.398267 21719 raft_consensus.cc:1275] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Refusing update from remote peer 3ecce86d24b04d3cae917bf243a4d819: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20251212 03:12:30.398514 21801 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20251212 03:12:30.398608 21773 consensus_queue.cc:1048] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20251212 03:12:30.399158 21776 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 1 LEADER]: Committing config change with OpId 1.35: config changed from index 34 to 35, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.399315 21567 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Committing config change with OpId 1.35: config changed from index 34 to 35, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.399323 21718 raft_consensus.cc:2955] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Committing config change with OpId 1.35: config changed from index 34 to 35, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.400033 20471 timestamp_advancement-itest.cc:267] GCing logs...
I20251212 03:12:30.400112 20471 log_reader.cc:385] T fe0a5b95bc99492e85e90eb6bdfe425b: removed 1 log segments from log reader
I20251212 03:12:30.400143 20471 log.cc:1079] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e: Deleting log segment in path: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765509114134682-20471-0/minicluster-data/ts-0-root/wals/fe0a5b95bc99492e85e90eb6bdfe425b/wal-000000001 (ops 1-15)
I20251212 03:12:30.400857 20471 timestamp_advancement-itest.cc:274] Shutting down cluster...
I20251212 03:12:30.400911 20471 master.cc:561] Master@127.19.253.254:46103 shutting down...
I20251212 03:12:30.400141 21465 catalog_manager.cc:5654] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e reported cstate change: config changed from index 34 to 35, 6718802d9d1b4ea0b2297512a373ad5e (127.19.253.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3ecce86d24b04d3cae917bf243a4d819" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } } }
I20251212 03:12:30.406477 21449 catalog_manager.cc:5201] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fe0a5b95bc99492e85e90eb6bdfe425b with cas_config_opid_index 24: aborting the task: latest config opid_index 35; task opid_index 24
I20251212 03:12:30.411248 21452 catalog_manager.cc:5201] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fe0a5b95bc99492e85e90eb6bdfe425b with cas_config_opid_index 26: aborting the task: latest config opid_index 35; task opid_index 26
I20251212 03:12:30.412420 21450 catalog_manager.cc:5201] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fe0a5b95bc99492e85e90eb6bdfe425b with cas_config_opid_index 22: aborting the task: latest config opid_index 35; task opid_index 22
I20251212 03:12:30.413549 21450 catalog_manager.cc:5201] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fe0a5b95bc99492e85e90eb6bdfe425b with cas_config_opid_index 20: aborting the task: latest config opid_index 35; task opid_index 20
I20251212 03:12:30.421936 21451 catalog_manager.cc:5201] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fe0a5b95bc99492e85e90eb6bdfe425b with cas_config_opid_index 34: aborting the task: latest config opid_index 35; task opid_index 34
I20251212 03:12:30.434511 21452 catalog_manager.cc:5201] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fe0a5b95bc99492e85e90eb6bdfe425b with cas_config_opid_index 28: aborting the task: latest config opid_index 35; task opid_index 28
I20251212 03:12:30.444133 21451 catalog_manager.cc:5201] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fe0a5b95bc99492e85e90eb6bdfe425b with cas_config_opid_index 30: aborting the task: latest config opid_index 35; task opid_index 30
I20251212 03:12:30.451763 21449 catalog_manager.cc:5201] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fe0a5b95bc99492e85e90eb6bdfe425b with cas_config_opid_index 32: aborting the task: latest config opid_index 35; task opid_index 32
I20251212 03:12:30.457212 20471 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e [term 1 LEADER]: Raft consensus shutting down.
I20251212 03:12:30.457309 20471 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251212 03:12:30.457363 20471 tablet_replica.cc:333] T 00000000000000000000000000000000 P 55fddf801b49476d88fa2f1a4842ce0e: stopping tablet replica
I20251212 03:12:30.470023 20471 master.cc:583] Master@127.19.253.254:46103 shutdown complete.
I20251212 03:12:30.471565 20471 tablet_server.cc:178] TabletServer@127.19.253.193:0 shutting down...
I20251212 03:12:30.482574 20471 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251212 03:12:30.482787 20471 tablet_replica.cc:333] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e: stopping tablet replica
I20251212 03:12:30.482861 20471 raft_consensus.cc:2243] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251212 03:12:30.482961 20471 raft_consensus.cc:2272] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251212 03:12:30.484892 20471 tablet_server.cc:195] TabletServer@127.19.253.193:0 shutdown complete.
I20251212 03:12:30.485810 20471 tablet_server.cc:178] TabletServer@127.19.253.194:0 shutting down...
I20251212 03:12:30.501639 20471 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251212 03:12:30.501821 20471 tablet_replica.cc:333] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819: stopping tablet replica
I20251212 03:12:30.501890 20471 raft_consensus.cc:2243] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 1 LEADER]: Raft consensus shutting down.
I20251212 03:12:30.501967 20471 raft_consensus.cc:2272] T fe0a5b95bc99492e85e90eb6bdfe425b P 3ecce86d24b04d3cae917bf243a4d819 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251212 03:12:30.504190 20471 tablet_server.cc:195] TabletServer@127.19.253.194:0 shutdown complete.
I20251212 03:12:30.504937 20471 tablet_server.cc:178] TabletServer@127.19.253.195:0 shutting down...
I20251212 03:12:30.526927 20471 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251212 03:12:30.527196 20471 tablet_replica.cc:333] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421: stopping tablet replica
I20251212 03:12:30.527249 20471 raft_consensus.cc:2243] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251212 03:12:30.527503 20471 raft_consensus.cc:2272] T fe0a5b95bc99492e85e90eb6bdfe425b P 9983e22759b64f91ade67eb78d5ce421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251212 03:12:30.539685 20471 tablet_server.cc:195] TabletServer@127.19.253.195:0 shutdown complete.
I20251212 03:12:30.540642 20471 timestamp_advancement-itest.cc:282] Restarting single tablet server...
I20251212 03:12:30.540751 20471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 03:12:30.542008 21802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:30.542186 21803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:30.542039 21805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:12:30.542271 20471 server_base.cc:1047] running on GCE node
I20251212 03:12:30.542429 20471 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251212 03:12:30.542469 20471 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251212 03:12:30.542482 20471 hybrid_clock.cc:648] HybridClock initialized: now 1765509150542482 us; error 0 us; skew 500 ppm
I20251212 03:12:30.543120 20471 webserver.cc:492] Webserver started at http://127.19.253.193:36677/ using document root <none> and password file <none>
I20251212 03:12:30.543202 20471 fs_manager.cc:362] Metadata directory not provided
I20251212 03:12:30.543241 20471 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251212 03:12:30.543919 20471 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:30.545200 21812 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:30.545359 20471 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251212 03:12:30.545418 20471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765509114134682-20471-0/minicluster-data/ts-0-root
uuid: "6718802d9d1b4ea0b2297512a373ad5e"
format_stamp: "Formatted at 2025-12-12 03:12:29 on dist-test-slave-q05w"
I20251212 03:12:30.545471 20471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765509114134682-20471-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765509114134682-20471-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1765509114134682-20471-0/minicluster-data/ts-0-root/data
Total live blocks: 5
Total live bytes: 5267
Total live bytes (after alignment): 24576
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 03:12:30.562844 20471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 03:12:30.563060 20471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 03:12:30.563244 20471 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251212 03:12:30.564049 20471 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251212 03:12:30.564090 20471 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251212 03:12:30.564551 20471 ts_tablet_manager.cc:616] Registered 1 tablets
I20251212 03:12:30.564587 20471 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:30.564605 21819 tablet_bootstrap.cc:492] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e: Bootstrap starting.
I20251212 03:12:30.566864 21819 tablet_bootstrap.cc:492] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251212 03:12:30.567118 21819 tablet_bootstrap.cc:492] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e: Bootstrap complete.
I20251212 03:12:30.567234 21819 ts_tablet_manager.cc:1403] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251212 03:12:30.567374 21819 raft_consensus.cc:359] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.567448 21819 raft_consensus.cc:740] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6718802d9d1b4ea0b2297512a373ad5e, State: Initialized, Role: FOLLOWER
I20251212 03:12:30.567507 21819 consensus_queue.cc:260] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 35, Last appended: 1.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "3ecce86d24b04d3cae917bf243a4d819" member_type: VOTER last_known_addr { host: "127.19.253.194" port: 43843 } } peers { permanent_uuid: "9983e22759b64f91ade67eb78d5ce421" member_type: VOTER last_known_addr { host: "127.19.253.195" port: 36039 } } peers { permanent_uuid: "6718802d9d1b4ea0b2297512a373ad5e" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 41509 } }
I20251212 03:12:30.567669 21819 ts_tablet_manager.cc:1434] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251212 03:12:30.568068 20471 rpc_server.cc:307] RPC server started. Bound to: 127.19.253.193:41509
I20251212 03:12:30.568634 21878 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.253.193:41509 every 8 connection(s)
W20251212 03:12:30.570070 21879 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251212 03:12:30.575788 21823 tablet_service.cc:3012] Rejecting scan request for tablet fe0a5b95bc99492e85e90eb6bdfe425b: Uninitialized: clean time has not yet been initialized
I20251212 03:12:30.576031 20471 timestamp_advancement-itest.cc:447] Got response: error { code: TABLET_NOT_RUNNING status { code: UNINITIALIZED message: "clean time has not yet been initialized" } }
I20251212 03:12:30.576344 20471 tablet_server.cc:178] TabletServer@127.19.253.193:41509 shutting down...
I20251212 03:12:30.819080 20471 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251212 03:12:30.819317 20471 tablet_replica.cc:333] stopping tablet replica
I20251212 03:12:30.819392 20471 raft_consensus.cc:2243] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251212 03:12:30.819453 20471 raft_consensus.cc:2272] T fe0a5b95bc99492e85e90eb6bdfe425b P 6718802d9d1b4ea0b2297512a373ad5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251212 03:12:30.831132 20471 tablet_server.cc:195] TabletServer@127.19.253.193:41509 shutdown complete.
[ OK ] TimestampAdvancementITest.Kudu2463Test (928 ms)
[ RUN ] TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp
I20251212 03:12:30.835819 20471 timestamp_advancement-itest.cc:460] Running with 1 replicas
I20251212 03:12:30.835980 20471 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.253.254:42809
I20251212 03:12:30.836122 20471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 03:12:30.837535 21884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:30.837692 21886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:30.837734 21883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:12:30.837838 20471 server_base.cc:1047] running on GCE node
I20251212 03:12:30.837940 20471 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251212 03:12:30.837965 20471 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251212 03:12:30.837977 20471 hybrid_clock.cc:648] HybridClock initialized: now 1765509150837978 us; error 0 us; skew 500 ppm
I20251212 03:12:30.838529 20471 webserver.cc:492] Webserver started at http://127.19.253.254:43251/ using document root <none> and password file <none>
I20251212 03:12:30.838593 20471 fs_manager.cc:362] Metadata directory not provided
I20251212 03:12:30.838626 20471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 03:12:30.838655 20471 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251212 03:12:30.838881 20471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/master-0-root/instance:
uuid: "baadf14028214d72b40e538cb02e1671"
format_stamp: "Formatted at 2025-12-12 03:12:30 on dist-test-slave-q05w"
I20251212 03:12:30.839722 20471 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251212 03:12:30.840128 21891 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:30.840263 20471 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:30.840312 20471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/master-0-root
uuid: "baadf14028214d72b40e538cb02e1671"
format_stamp: "Formatted at 2025-12-12 03:12:30 on dist-test-slave-q05w"
I20251212 03:12:30.840348 20471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 03:12:30.858489 20471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 03:12:30.858667 20471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 03:12:30.861115 20471 rpc_server.cc:307] RPC server started. Bound to: 127.19.253.254:42809
I20251212 03:12:30.861783 21943 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.253.254:42809 every 8 connection(s)
I20251212 03:12:30.861902 21944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 03:12:30.862920 21944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671: Bootstrap starting.
I20251212 03:12:30.863207 21944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671: Neither blocks nor log segments found. Creating new log.
I20251212 03:12:30.863751 21944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671: No bootstrap required, opened a new log
I20251212 03:12:30.863873 21944 raft_consensus.cc:359] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baadf14028214d72b40e538cb02e1671" member_type: VOTER }
I20251212 03:12:30.863940 21944 raft_consensus.cc:385] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 03:12:30.863955 21944 raft_consensus.cc:740] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baadf14028214d72b40e538cb02e1671, State: Initialized, Role: FOLLOWER
I20251212 03:12:30.863991 21944 consensus_queue.cc:260] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baadf14028214d72b40e538cb02e1671" member_type: VOTER }
I20251212 03:12:30.864054 21944 raft_consensus.cc:399] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251212 03:12:30.864080 21944 raft_consensus.cc:493] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251212 03:12:30.864105 21944 raft_consensus.cc:3060] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [term 0 FOLLOWER]: Advancing to term 1
I20251212 03:12:30.864624 21944 raft_consensus.cc:515] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baadf14028214d72b40e538cb02e1671" member_type: VOTER }
I20251212 03:12:30.864682 21944 leader_election.cc:304] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baadf14028214d72b40e538cb02e1671; no voters:
I20251212 03:12:30.864777 21944 leader_election.cc:290] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251212 03:12:30.864866 21947 raft_consensus.cc:2804] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [term 1 FOLLOWER]: Leader election won for term 1
I20251212 03:12:30.864912 21944 sys_catalog.cc:565] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [sys.catalog]: configured and running, proceeding with master startup.
I20251212 03:12:30.865005 21947 raft_consensus.cc:697] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [term 1 LEADER]: Becoming Leader. State: Replica: baadf14028214d72b40e538cb02e1671, State: Running, Role: LEADER
I20251212 03:12:30.865059 21947 consensus_queue.cc:237] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baadf14028214d72b40e538cb02e1671" member_type: VOTER }
I20251212 03:12:30.865258 21948 sys_catalog.cc:455] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "baadf14028214d72b40e538cb02e1671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baadf14028214d72b40e538cb02e1671" member_type: VOTER } }
I20251212 03:12:30.865319 21948 sys_catalog.cc:458] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [sys.catalog]: This master's current role is: LEADER
I20251212 03:12:30.865478 21953 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251212 03:12:30.865463 21949 sys_catalog.cc:455] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [sys.catalog]: SysCatalogTable state changed. Reason: New leader baadf14028214d72b40e538cb02e1671. Latest consensus state: current_term: 1 leader_uuid: "baadf14028214d72b40e538cb02e1671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baadf14028214d72b40e538cb02e1671" member_type: VOTER } }
I20251212 03:12:30.865739 21949 sys_catalog.cc:458] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [sys.catalog]: This master's current role is: LEADER
I20251212 03:12:30.865826 21953 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251212 03:12:30.866312 20471 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251212 03:12:30.866415 21953 catalog_manager.cc:1357] Generated new cluster ID: 8bee50bfeb1c404e890a0d0a75d9e7d8
I20251212 03:12:30.866465 21953 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251212 03:12:30.889274 21953 catalog_manager.cc:1380] Generated new certificate authority record
I20251212 03:12:30.889742 21953 catalog_manager.cc:1514] Loading token signing keys...
I20251212 03:12:30.896207 21953 catalog_manager.cc:6027] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671: Generated new TSK 0
I20251212 03:12:30.896292 21953 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251212 03:12:30.898101 20471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 03:12:30.899538 21966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:30.899538 21968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:30.899610 21965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:12:30.899760 20471 server_base.cc:1047] running on GCE node
I20251212 03:12:30.899896 20471 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251212 03:12:30.899930 20471 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251212 03:12:30.899942 20471 hybrid_clock.cc:648] HybridClock initialized: now 1765509150899943 us; error 0 us; skew 500 ppm
I20251212 03:12:30.900521 20471 webserver.cc:492] Webserver started at http://127.19.253.193:44757/ using document root <none> and password file <none>
I20251212 03:12:30.900591 20471 fs_manager.cc:362] Metadata directory not provided
I20251212 03:12:30.900628 20471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 03:12:30.900666 20471 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251212 03:12:30.900894 20471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/ts-0-root/instance:
uuid: "55b35efdf6f94f0dbf6038719c58e238"
format_stamp: "Formatted at 2025-12-12 03:12:30 on dist-test-slave-q05w"
I20251212 03:12:30.901746 20471 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251212 03:12:30.902177 21973 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:30.902316 20471 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:30.902364 20471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/ts-0-root
uuid: "55b35efdf6f94f0dbf6038719c58e238"
format_stamp: "Formatted at 2025-12-12 03:12:30 on dist-test-slave-q05w"
I20251212 03:12:30.902407 20471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 03:12:30.909744 20471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 03:12:30.909917 20471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 03:12:30.910048 20471 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251212 03:12:30.910288 20471 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251212 03:12:30.910323 20471 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:30.910362 20471 ts_tablet_manager.cc:616] Registered 0 tablets
I20251212 03:12:30.910388 20471 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:30.913107 20471 rpc_server.cc:307] RPC server started. Bound to: 127.19.253.193:35973
I20251212 03:12:30.913159 22036 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.253.193:35973 every 8 connection(s)
I20251212 03:12:30.916604 22037 heartbeater.cc:344] Connected to a master server at 127.19.253.254:42809
I20251212 03:12:30.916671 22037 heartbeater.cc:461] Registering TS with master...
I20251212 03:12:30.916777 22037 heartbeater.cc:507] Master 127.19.253.254:42809 requested a full tablet report, sending...
I20251212 03:12:30.917382 21908 ts_manager.cc:194] Registered new tserver with Master: 55b35efdf6f94f0dbf6038719c58e238 (127.19.253.193:35973)
I20251212 03:12:30.917877 20471 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004606602s
I20251212 03:12:30.918232 21908 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60548
I20251212 03:12:30.924665 20471 test_util.cc:276] Using random seed: -1580605576
I20251212 03:12:30.928807 21908 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60566:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251212 03:12:30.932633 22001 tablet_service.cc:1505] Processing CreateTablet for tablet 2ca36359da71475595e759311d140211 (DEFAULT_TABLE table=test-workload [id=a784c1f582f540bf898d40596f58e453]), partition=RANGE (key) PARTITION UNBOUNDED
I20251212 03:12:30.932739 22001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ca36359da71475595e759311d140211. 1 dirs total, 0 dirs full, 0 dirs failed
I20251212 03:12:30.933935 22055 tablet_bootstrap.cc:492] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238: Bootstrap starting.
I20251212 03:12:30.934258 22055 tablet_bootstrap.cc:654] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238: Neither blocks nor log segments found. Creating new log.
I20251212 03:12:30.934713 22055 tablet_bootstrap.cc:492] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238: No bootstrap required, opened a new log
I20251212 03:12:30.934757 22055 ts_tablet_manager.cc:1403] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251212 03:12:30.934854 22055 raft_consensus.cc:359] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b35efdf6f94f0dbf6038719c58e238" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 35973 } }
I20251212 03:12:30.934897 22055 raft_consensus.cc:385] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251212 03:12:30.934921 22055 raft_consensus.cc:740] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55b35efdf6f94f0dbf6038719c58e238, State: Initialized, Role: FOLLOWER
I20251212 03:12:30.934980 22055 consensus_queue.cc:260] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b35efdf6f94f0dbf6038719c58e238" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 35973 } }
I20251212 03:12:30.935014 22055 raft_consensus.cc:399] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251212 03:12:30.935037 22055 raft_consensus.cc:493] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251212 03:12:30.935067 22055 raft_consensus.cc:3060] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [term 0 FOLLOWER]: Advancing to term 1
I20251212 03:12:30.935588 22055 raft_consensus.cc:515] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b35efdf6f94f0dbf6038719c58e238" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 35973 } }
I20251212 03:12:30.935650 22055 leader_election.cc:304] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55b35efdf6f94f0dbf6038719c58e238; no voters:
I20251212 03:12:30.935761 22055 leader_election.cc:290] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251212 03:12:30.935817 22057 raft_consensus.cc:2804] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [term 1 FOLLOWER]: Leader election won for term 1
I20251212 03:12:30.935915 22055 ts_tablet_manager.cc:1434] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251212 03:12:30.935989 22057 raft_consensus.cc:697] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [term 1 LEADER]: Becoming Leader. State: Replica: 55b35efdf6f94f0dbf6038719c58e238, State: Running, Role: LEADER
I20251212 03:12:30.936025 22037 heartbeater.cc:499] Master 127.19.253.254:42809 was elected leader, sending a full tablet report...
I20251212 03:12:30.936045 22057 consensus_queue.cc:237] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b35efdf6f94f0dbf6038719c58e238" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 35973 } }
I20251212 03:12:30.936718 21908 catalog_manager.cc:5654] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55b35efdf6f94f0dbf6038719c58e238 (127.19.253.193). New cstate: current_term: 1 leader_uuid: "55b35efdf6f94f0dbf6038719c58e238" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b35efdf6f94f0dbf6038719c58e238" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 35973 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:30.938861 20471 tablet_server.cc:178] TabletServer@127.19.253.193:0 shutting down...
I20251212 03:12:30.941354 20471 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251212 03:12:30.941537 20471 tablet_replica.cc:333] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238: stopping tablet replica
I20251212 03:12:30.941593 20471 raft_consensus.cc:2243] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [term 1 LEADER]: Raft consensus shutting down.
I20251212 03:12:30.941637 20471 raft_consensus.cc:2272] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251212 03:12:30.953502 20471 tablet_server.cc:195] TabletServer@127.19.253.193:0 shutdown complete.
I20251212 03:12:30.954445 20471 master.cc:561] Master@127.19.253.254:42809 shutting down...
I20251212 03:12:30.956765 20471 raft_consensus.cc:2243] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [term 1 LEADER]: Raft consensus shutting down.
I20251212 03:12:30.956822 20471 raft_consensus.cc:2272] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251212 03:12:30.956885 20471 tablet_replica.cc:333] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671: stopping tablet replica
I20251212 03:12:30.968813 20471 master.cc:583] Master@127.19.253.254:42809 shutdown complete.
I20251212 03:12:30.970631 20471 timestamp_advancement-itest.cc:460] Running with 3 replicas
I20251212 03:12:30.970769 20471 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.19.253.254:36855
I20251212 03:12:30.970878 20471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 03:12:30.971982 22063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:30.972106 22062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:30.972162 22065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:12:30.972263 20471 server_base.cc:1047] running on GCE node
I20251212 03:12:30.972358 20471 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251212 03:12:30.972389 20471 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251212 03:12:30.972406 20471 hybrid_clock.cc:648] HybridClock initialized: now 1765509150972406 us; error 0 us; skew 500 ppm
I20251212 03:12:30.972957 20471 webserver.cc:492] Webserver started at http://127.19.253.254:36503/ using document root <none> and password file <none>
I20251212 03:12:30.973027 20471 fs_manager.cc:362] Metadata directory not provided
I20251212 03:12:30.973074 20471 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251212 03:12:30.973618 20471 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:30.974009 22070 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:30.974159 20471 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:30.974218 20471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/master-0-root
uuid: "baadf14028214d72b40e538cb02e1671"
format_stamp: "Formatted at 2025-12-12 03:12:30 on dist-test-slave-q05w"
I20251212 03:12:30.974303 20471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 03:12:30.984930 20471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 03:12:30.985113 20471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 03:12:30.987670 20471 rpc_server.cc:307] RPC server started. Bound to: 127.19.253.254:36855
I20251212 03:12:30.988641 22122 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.253.254:36855 every 8 connection(s)
I20251212 03:12:30.989180 22123 sys_catalog.cc:263] Verifying existing consensus state
I20251212 03:12:30.989506 22123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671: Bootstrap starting.
I20251212 03:12:30.990968 22123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251212 03:12:30.991169 22123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671: Bootstrap complete.
I20251212 03:12:30.991341 22123 raft_consensus.cc:359] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baadf14028214d72b40e538cb02e1671" member_type: VOTER }
I20251212 03:12:30.991407 22123 raft_consensus.cc:740] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: baadf14028214d72b40e538cb02e1671, State: Initialized, Role: FOLLOWER
I20251212 03:12:30.991451 22123 consensus_queue.cc:260] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baadf14028214d72b40e538cb02e1671" member_type: VOTER }
I20251212 03:12:30.991487 22123 raft_consensus.cc:399] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251212 03:12:30.991505 22123 raft_consensus.cc:493] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251212 03:12:30.991523 22123 raft_consensus.cc:3060] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [term 1 FOLLOWER]: Advancing to term 2
I20251212 03:12:30.991997 22123 raft_consensus.cc:515] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baadf14028214d72b40e538cb02e1671" member_type: VOTER }
I20251212 03:12:30.992049 22123 leader_election.cc:304] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baadf14028214d72b40e538cb02e1671; no voters:
I20251212 03:12:30.992132 22123 leader_election.cc:290] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [CANDIDATE]: Term 2 election: Requested vote from peers
I20251212 03:12:30.992209 22126 raft_consensus.cc:2804] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [term 2 FOLLOWER]: Leader election won for term 2
I20251212 03:12:30.992285 22123 sys_catalog.cc:565] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [sys.catalog]: configured and running, proceeding with master startup.
I20251212 03:12:30.992329 22126 raft_consensus.cc:697] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [term 2 LEADER]: Becoming Leader. State: Replica: baadf14028214d72b40e538cb02e1671, State: Running, Role: LEADER
I20251212 03:12:30.992386 22126 consensus_queue.cc:237] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baadf14028214d72b40e538cb02e1671" member_type: VOTER }
I20251212 03:12:30.992619 22128 sys_catalog.cc:455] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [sys.catalog]: SysCatalogTable state changed. Reason: New leader baadf14028214d72b40e538cb02e1671. Latest consensus state: current_term: 2 leader_uuid: "baadf14028214d72b40e538cb02e1671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baadf14028214d72b40e538cb02e1671" member_type: VOTER } }
I20251212 03:12:30.992690 22128 sys_catalog.cc:458] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [sys.catalog]: This master's current role is: LEADER
I20251212 03:12:30.992630 22127 sys_catalog.cc:455] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "baadf14028214d72b40e538cb02e1671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baadf14028214d72b40e538cb02e1671" member_type: VOTER } }
I20251212 03:12:30.992748 22127 sys_catalog.cc:458] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [sys.catalog]: This master's current role is: LEADER
I20251212 03:12:30.992982 22131 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251212 03:12:30.993259 22131 catalog_manager.cc:679] Loaded metadata for table test-workload [id=a784c1f582f540bf898d40596f58e453]
I20251212 03:12:30.993391 22131 tablet_loader.cc:96] loaded metadata for tablet 2ca36359da71475595e759311d140211 (table test-workload [id=a784c1f582f540bf898d40596f58e453])
I20251212 03:12:30.993441 22131 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251212 03:12:30.993521 22131 catalog_manager.cc:1269] Loaded cluster ID: 8bee50bfeb1c404e890a0d0a75d9e7d8
I20251212 03:12:30.993552 22131 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251212 03:12:30.993582 20471 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251212 03:12:30.994498 22131 catalog_manager.cc:1514] Loading token signing keys...
I20251212 03:12:30.994616 22131 catalog_manager.cc:6038] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671: Loaded TSK: 0
I20251212 03:12:30.994725 22131 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251212 03:12:30.994938 20471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251212 03:12:30.996276 20471 server_base.cc:1047] running on GCE node
W20251212 03:12:30.996304 22144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:30.996255 22143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:30.996400 22146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:12:30.996727 20471 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251212 03:12:30.996762 20471 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251212 03:12:30.996783 20471 hybrid_clock.cc:648] HybridClock initialized: now 1765509150996782 us; error 0 us; skew 500 ppm
I20251212 03:12:30.997308 20471 webserver.cc:492] Webserver started at http://127.19.253.193:35161/ using document root <none> and password file <none>
I20251212 03:12:30.997395 20471 fs_manager.cc:362] Metadata directory not provided
I20251212 03:12:30.997442 20471 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251212 03:12:30.997994 20471 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20251212 03:12:30.998476 22151 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:30.998591 20471 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:30.998632 20471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/ts-0-root
uuid: "55b35efdf6f94f0dbf6038719c58e238"
format_stamp: "Formatted at 2025-12-12 03:12:30 on dist-test-slave-q05w"
I20251212 03:12:30.998687 20471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 03:12:31.015916 20471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 03:12:31.016165 20471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 03:12:31.016328 20471 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251212 03:12:31.016619 22158 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20251212 03:12:31.017198 20471 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251212 03:12:31.017246 20471 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251212 03:12:31.017287 20471 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20251212 03:12:31.017701 20471 ts_tablet_manager.cc:616] Registered 1 tablets
I20251212 03:12:31.017730 20471 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:31.017774 22158 tablet_bootstrap.cc:492] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238: Bootstrap starting.
I20251212 03:12:31.019241 22158 tablet_bootstrap.cc:492] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251212 03:12:31.019507 22158 tablet_bootstrap.cc:492] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238: Bootstrap complete.
I20251212 03:12:31.019618 22158 ts_tablet_manager.cc:1403] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251212 03:12:31.019748 22158 raft_consensus.cc:359] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b35efdf6f94f0dbf6038719c58e238" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 35973 } }
I20251212 03:12:31.019824 22158 raft_consensus.cc:740] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55b35efdf6f94f0dbf6038719c58e238, State: Initialized, Role: FOLLOWER
I20251212 03:12:31.019876 22158 consensus_queue.cc:260] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b35efdf6f94f0dbf6038719c58e238" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 35973 } }
I20251212 03:12:31.019923 22158 raft_consensus.cc:399] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251212 03:12:31.019954 22158 raft_consensus.cc:493] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251212 03:12:31.019982 22158 raft_consensus.cc:3060] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [term 1 FOLLOWER]: Advancing to term 2
I20251212 03:12:31.020879 22158 raft_consensus.cc:515] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b35efdf6f94f0dbf6038719c58e238" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 35973 } }
I20251212 03:12:31.020953 22158 leader_election.cc:304] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55b35efdf6f94f0dbf6038719c58e238; no voters:
I20251212 03:12:31.021080 22158 leader_election.cc:290] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [CANDIDATE]: Term 2 election: Requested vote from peers
I20251212 03:12:31.021124 20471 rpc_server.cc:307] RPC server started. Bound to: 127.19.253.193:38933
I20251212 03:12:31.021173 22216 raft_consensus.cc:2804] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [term 2 FOLLOWER]: Leader election won for term 2
I20251212 03:12:31.021219 22158 ts_tablet_manager.cc:1434] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251212 03:12:31.021270 22216 raft_consensus.cc:697] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [term 2 LEADER]: Becoming Leader. State: Replica: 55b35efdf6f94f0dbf6038719c58e238, State: Running, Role: LEADER
I20251212 03:12:31.021343 22216 consensus_queue.cc:237] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b35efdf6f94f0dbf6038719c58e238" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 35973 } }
I20251212 03:12:31.021277 22217 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.253.193:38933 every 8 connection(s)
I20251212 03:12:31.021522 20471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251212 03:12:31.022786 22226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:31.022825 22224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:12:31.023017 20471 server_base.cc:1047] running on GCE node
W20251212 03:12:31.023289 22229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:12:31.023439 20471 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251212 03:12:31.023486 20471 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251212 03:12:31.023509 20471 hybrid_clock.cc:648] HybridClock initialized: now 1765509151023507 us; error 0 us; skew 500 ppm
I20251212 03:12:31.024202 20471 webserver.cc:492] Webserver started at http://127.19.253.194:42117/ using document root <none> and password file <none>
I20251212 03:12:31.024294 20471 fs_manager.cc:362] Metadata directory not provided
I20251212 03:12:31.024362 20471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 03:12:31.024405 20471 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251212 03:12:31.024712 20471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/ts-1-root/instance:
uuid: "42c0b4bbf9d7432e84e34dd028055b28"
format_stamp: "Formatted at 2025-12-12 03:12:31 on dist-test-slave-q05w"
I20251212 03:12:31.025552 22220 heartbeater.cc:344] Connected to a master server at 127.19.253.254:36855
I20251212 03:12:31.025619 22220 heartbeater.cc:461] Registering TS with master...
I20251212 03:12:31.025714 22220 heartbeater.cc:507] Master 127.19.253.254:36855 requested a full tablet report, sending...
I20251212 03:12:31.025734 20471 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251212 03:12:31.025980 22087 ts_manager.cc:194] Registered new tserver with Master: 55b35efdf6f94f0dbf6038719c58e238 (127.19.253.193:38933)
I20251212 03:12:31.026051 22087 catalog_manager.cc:5654] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "55b35efdf6f94f0dbf6038719c58e238" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55b35efdf6f94f0dbf6038719c58e238" member_type: VOTER last_known_addr { host: "127.19.253.193" port: 35973 } health_report { overall_health: HEALTHY } } }
I20251212 03:12:31.026274 22234 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:31.026407 20471 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:31.026454 20471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/ts-1-root
uuid: "42c0b4bbf9d7432e84e34dd028055b28"
format_stamp: "Formatted at 2025-12-12 03:12:31 on dist-test-slave-q05w"
I20251212 03:12:31.026496 20471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 03:12:31.027215 22087 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49424
I20251212 03:12:31.027613 22220 heartbeater.cc:499] Master 127.19.253.254:36855 was elected leader, sending a full tablet report...
I20251212 03:12:31.040238 20471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 03:12:31.040511 20471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 03:12:31.040688 20471 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251212 03:12:31.040894 20471 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251212 03:12:31.040922 20471 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:31.040958 20471 ts_tablet_manager.cc:616] Registered 0 tablets
I20251212 03:12:31.040984 20471 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:31.044224 20471 rpc_server.cc:307] RPC server started. Bound to: 127.19.253.194:39319
I20251212 03:12:31.044305 22298 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.253.194:39319 every 8 connection(s)
I20251212 03:12:31.044590 20471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251212 03:12:31.046152 20471 server_base.cc:1047] running on GCE node
W20251212 03:12:31.046253 22305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:31.046139 22302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251212 03:12:31.047519 22303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251212 03:12:31.047751 20471 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251212 03:12:31.047801 20471 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251212 03:12:31.047825 20471 hybrid_clock.cc:648] HybridClock initialized: now 1765509151047825 us; error 0 us; skew 500 ppm
I20251212 03:12:31.047943 22299 heartbeater.cc:344] Connected to a master server at 127.19.253.254:36855
I20251212 03:12:31.048007 22299 heartbeater.cc:461] Registering TS with master...
I20251212 03:12:31.048127 22299 heartbeater.cc:507] Master 127.19.253.254:36855 requested a full tablet report, sending...
I20251212 03:12:31.048400 22087 ts_manager.cc:194] Registered new tserver with Master: 42c0b4bbf9d7432e84e34dd028055b28 (127.19.253.194:39319)
I20251212 03:12:31.048514 20471 webserver.cc:492] Webserver started at http://127.19.253.195:35411/ using document root <none> and password file <none>
I20251212 03:12:31.048607 20471 fs_manager.cc:362] Metadata directory not provided
I20251212 03:12:31.048655 20471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251212 03:12:31.048705 20471 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251212 03:12:31.048923 22087 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49428
I20251212 03:12:31.048956 20471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/ts-2-root/instance:
uuid: "287a8685504e45c7bb45cfc3f392e204"
format_stamp: "Formatted at 2025-12-12 03:12:31 on dist-test-slave-q05w"
I20251212 03:12:31.049858 20471 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20251212 03:12:31.050405 22310 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:31.050542 20471 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:31.050587 20471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/ts-2-root
uuid: "287a8685504e45c7bb45cfc3f392e204"
format_stamp: "Formatted at 2025-12-12 03:12:31 on dist-test-slave-q05w"
I20251212 03:12:31.050638 20471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskieiyXB/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1765509114134682-20471-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251212 03:12:31.062922 20471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251212 03:12:31.063174 20471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251212 03:12:31.063351 20471 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251212 03:12:31.063551 20471 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251212 03:12:31.063581 20471 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:31.063614 20471 ts_tablet_manager.cc:616] Registered 0 tablets
I20251212 03:12:31.063643 20471 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251212 03:12:31.066908 20471 rpc_server.cc:307] RPC server started. Bound to: 127.19.253.195:40279
I20251212 03:12:31.066968 22373 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.19.253.195:40279 every 8 connection(s)
I20251212 03:12:31.070151 22374 heartbeater.cc:344] Connected to a master server at 127.19.253.254:36855
I20251212 03:12:31.070250 22374 heartbeater.cc:461] Registering TS with master...
I20251212 03:12:31.070356 22374 heartbeater.cc:507] Master 127.19.253.254:36855 requested a full tablet report, sending...
I20251212 03:12:31.070667 22087 ts_manager.cc:194] Registered new tserver with Master: 287a8685504e45c7bb45cfc3f392e204 (127.19.253.195:40279)
I20251212 03:12:31.070991 22087 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49440
I20251212 03:12:31.071595 20471 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00451115s
I20251212 03:12:31.079129 20471 test_util.cc:276] Using random seed: -1580451111
I20251212 03:12:31.083992 20471 test_workload.cc:405] TestWorkload: Skipping table creation because table test-workload already exists
I20251212 03:12:31.084064 20471 tablet_server.cc:178] TabletServer@127.19.253.193:0 shutting down...
I20251212 03:12:31.086246 20471 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251212 03:12:31.086475 20471 tablet_replica.cc:333] stopping tablet replica
I20251212 03:12:31.086534 20471 raft_consensus.cc:2243] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [term 2 LEADER]: Raft consensus shutting down.
I20251212 03:12:31.086589 20471 raft_consensus.cc:2272] T 2ca36359da71475595e759311d140211 P 55b35efdf6f94f0dbf6038719c58e238 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251212 03:12:31.098630 20471 tablet_server.cc:195] TabletServer@127.19.253.193:0 shutdown complete.
I20251212 03:12:31.099639 20471 tablet_server.cc:178] TabletServer@127.19.253.194:0 shutting down...
I20251212 03:12:31.102116 20471 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251212 03:12:31.114027 20471 tablet_server.cc:195] TabletServer@127.19.253.194:0 shutdown complete.
I20251212 03:12:31.114982 20471 tablet_server.cc:178] TabletServer@127.19.253.195:0 shutting down...
I20251212 03:12:31.118005 20471 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251212 03:12:31.129469 20471 tablet_server.cc:195] TabletServer@127.19.253.195:0 shutdown complete.
I20251212 03:12:31.130571 20471 master.cc:561] Master@127.19.253.254:36855 shutting down...
I20251212 03:12:31.133045 20471 raft_consensus.cc:2243] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [term 2 LEADER]: Raft consensus shutting down.
I20251212 03:12:31.133117 20471 raft_consensus.cc:2272] T 00000000000000000000000000000000 P baadf14028214d72b40e538cb02e1671 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251212 03:12:31.133188 20471 tablet_replica.cc:333] stopping tablet replica
I20251212 03:12:31.134939 20471 master.cc:583] Master@127.19.253.254:36855 shutdown complete.
[ OK ] TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp (303 ms)
[----------] 4 tests from TimestampAdvancementITest (36999 ms total)
[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (36999 ms total)
[ PASSED ] 3 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData
1 FAILED TEST
I20251212 03:12:31.139173 20471 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/catalog_manager.cc:5332: suppressed but not reported on 9 messages since previous log ~0 seconds ago
I20251212 03:12:31.139217 20471 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 9 messages since previous log ~1 seconds ago
I20251212 03:12:31.139246 20471 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 9 messages since previous log ~4 seconds ago
I20251212 03:12:31.139303 20471 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/maintenance_manager.cc:346: suppressed but not reported on 568 messages since previous log ~36 seconds ago